Logging arbitrary data
So far the tutorial and documentation have shown how logworm automatically records information about web requests, and explained how to query and process that data. But logging web requests is NOT the unique (nor the main) purpose of logworm. What tells logworm appart from other services is that it enables the developer to log any kind of application-specific data. With it, you can create arbitrary log tables (groups of log entries related to a particular topic), and have your application send schema-free, dynamic log entries for storage and later processing.
Logging data
Once your application requires the logworm_amqp gem, it can use the log_in method anywhere to record log entries. The syntax is very simple: you must pass the name of the logging table where the entry should be filed, and an arbitrary hashmap with key-value pairs, one for each piece of information that you want to record about the action.
For example, let's say that you want to add audit capabilities to your application. You first have to decide which events in your application (e.g., which actions on which controllers, if you're doing Rails) should be recorded; then, it's just a matter of calling log_in to log onto an audit table (let's call it audit) with the necessary information. For example, you'd write
log_in(:audit, {:action => "Logged in", :subject => current_user.email})log_in(:audit, {:action => "Created page", :page => @page.title, :subject => current_user.email})log_in(:audit, {:action => "Deleted category", category => @category.title, :subject => current_user.email})Notice how the schema for teh audit log table is free: some events contain a field named page, others contain instead a field named category, and yet others don't contain any special fields other than the action and the subject. This is a distinctive feature of logworm: log tables are always schema-less, to give developers complete flexibility when recording information about the functioning of their app. In the same way, log tables are created automatically if they don't exist, so it's really just a matter of adding a call to log_in whenever you want to record some information --no need to manually create tables, manage your account, etc.
One important corollary of the schema-less nature of log tables, by the way, is that you can edit your code at any time and change the fields that you log, and logworm will accept the new entries without a problem. Let's say for example that you decide to also record the category of the page that was created; you just have to edit the appropriate call and add the new field:
log_in(:audit, {:action => "Created page",
:page => @page.title, :category => @page.category,
:subject => current_user.email})logworm will record and index all this information, so you can later query it. Say, for example, that you want to get a list of the latest actions performed by a particular user. You could use lw-tail for that:
$ lw-tail audit -c '"subject":"schapira@pomelollc.com"
2010-03-02 @ 15:39:38 GMT ==>
_request_id: "23949566400260-1267544378088"
_ts_utc: 1267544378089
action: "Logged in"
2010-03-02 @ 15:40:40 GMT ==>
_request_id: "23949566400260-1267544380865"
_ts_utc: 1267544380866
action: "Created page"
page: "Welcome to logworm"
2010-03-02 @ 15:39:40 GMT ==>
_request_id: "23949566400260-1267544380865"
_ts_utc: 1267544382866
action: "Created page"
page: "How to use logworm"
category: "Tutorials"
IMPORTANT: If you're testing these new commands to log arbitrary data, please remember to reload the corresponding pages from your application to make sure that the logging command actually runs and creates the audit log table. Otherwise, lw-tail will complain about the table not existing.
If you're running your application locally, make sure that it's in production mode, as logworm does not log data in development mode.
Other examples of data that you may want to log
Anything that your application does is a good candidate for logging. In general our approach is "log everything first, ask questions later". Typical examples include:
- Audit information, as in the example above
- Security-related information
log_in(:logins, {:email => user.email, :successful => false})log_in(:oauth, {:action => :grant_token, :consumer => @app.title, :for => @user.email})log_in(:performance, {:type => :query, :table => "users", :query => :find_friends, :time => 0.3})log_in(:exceptions, {:controller => controller_name, :action => action_name, :type => exc.class, :dump => e.backtrace})log_in(:accounts, {:action => :create, :state => @user.state, :gender => @user.gender})log_in(:readership, {:action => :subscribe, :feed => @feed.url})log_in(:searches, {:text => search.text, :results => search.rows.count })log_in(:cart_use, {:action => :add_item, :item => item.id, :size => cart.count })
log_in(:cart_use, {:action => :remove_item, :item => item.id, :size => cart.count })
log_in(:cart_use, {:action => :init_checkout, :size => cart.count })
log_in(:cart_use, {:action => :checkout_complete })Some of this information may be also stored in your site's database, at least temporarily. But the key here is that while your database is used to model and store the data that your application needs to function, logworm logs are used to model and store information about how your application behaves. A typical case to highlight the difference has to do with account information: while your application may delete records from its users table as accounts are closed, you don't want to lose information about the fact that the accounts existed at some point in time --without that information you couldn't understand, for example, how the number of accounts in your system fluctuated over time.
More importantly, the idea behind thinking about logging as separate from your data model and store is that you never know what information you will need in the future. You obviously cannot contaminate your store with data just in case you might need it in the future, but that's precisely the purpose of a logging service: you store now, and ask questions later. Logging becomes an orthogonal aspect in your code, and logworm is built to make that aspect as easy to implement as possible.
Getting the most out of your log tables
With logworm you can log arbitrary data --whatever is relevant to your application and/or your business. But customized data requires customized tools to extract business-dependent knowledge from it. That's why the entire architecture of logworm is built around the notion of independent analysis tools: the API and the powerful query language are the foundation upon which small, custom applications can be built for specific purposes. There is a mechanism to grant permission for an application (either something that we build, or something that third-parties may build on top of our architecture) to access your data and give you results and reports, or even perform actions based on the results. For example, we are working on a very simple app that periodically queries the web_log table and detects an unusual increase in the size of the HEROKU_QUEUE values and automatically spawns a new Heroku instance while it's needed. Very small and very simple app, made possible thanks to the logworm API and the fact that the data is stored and analyzed.
At the time of launch we have command-line tools such as lw-tail and lw-compute, and a web-based console to run arbitrary queries. Our next step is to work on more specialized applications, that work best for certain kinds of log tables. In the works is our first "specialized" application: a tool to create RSS feeds from queries against a log entry. With it you will be able, for example, to offer an RSS feed that allows you to keep track of all the audit-related events in your system and, perhaps more interestingly, to embed those RSS feeds directly into your application and offer them to your users.
Back to Querying with the web-based console or On to the logworm API
Company | FAQ | Documentation | Blog
