Frequently Asked Questions about logworm

How is logworm different from Google Analytics and Mixpanel?

Unlike Google Analytics, which emphasizes pageview tracking, and Mixpanel, which focuses on A/B testing and funnel analyses, logworm is built around an API and a query language. Mixpanel can show you what's happening with your applications; so can logworm. Both record and analyze custom events. But unlike Mixpanel, which displays its analyses in mostly predetermined ways, logworm is completely flexible and open. It provides the platform and tools you need to do virtually anything you want with your data. You can visualize it, of course, but more importantly, you can take action. With logworm's API and minty fresh query language, you can write scripts that respond to any events you decide to record, and any queries you choose to run.

In addition, logworm is a Heroku add-on, making it a natural choice for tracking events generated by a Heroku application.

How is logworm different from Exceptional, New Relic, etc?

Again, the main difference is that logworm's purpose is to allow you to log, store, analyze, and act upon arbitrary kinds of data, anything that is relevant to your application and/or your business. Exceptional instead is a great tool for keeping track of exceptions in your application, while New Relic focuses more on performance data. More importantly, they have a closed, static data model, and only allow you to view your data through their reports; with logworm, you can log anything, and you can get it all back via our API and query language.

How is logworm different from Splunk?

Splunk can do everything logworm can do and more, but at a far higher price and requiring a much more complicated setup.

Can I use logworm with Mixpanel and / or Google Analytics at the same time?

Yes.

How does logworm work?

You simply have to add a gem in your application, and a configuration file with security keys identifying your application. Then you make calls to logworm throughout your application, to record the events you care about. The logworm gem collects that information locally and, when your application is about to finish the processing of its HTTP request, logworm establishes a connection and sends those data to our servers.

We store the data on our servers, and give you a powerful API and query language to retrieve it. On top of that API and query language, we have also built applications that give you reports, charts, alerts, and RSS feeds from your log entries. You get all that functionality right out-of-the-box, but since the API and the query language are open, you can always write custom applications on top of logworm.

Will logworm affect the performance of my application?

Yes. For every request that your application processes, logworm sends logging information to our servers. Once this process is complete, logworm calls home and sends all the logging information that you've recorded in the context of that HTTP request. Because this call home is performed before the connection is closed, it does impact your application's performance.

The impact is minimal, however. All our servers (Heroku applications themselves!) are in the same network as your Heroku application, which means that the network roundtrips are very quick. In normal conditions, logworm will add between 10ms to 40ms to the processing time of your request. This is mostly a matter of network latency, as we only do basic security checks when we receive your request, and process it later in the background.

To make things even faster, we are currently working with Heroku to let logworm clients communicate with our servers with a superfast AMQP service. Once AMQP comes out of private beta in Heroku, we will use it as our default method of sending data to our servers.

Where and how are you storing my data?

We store your data with MongoHQ, a cloud-based hosted database solution. They use the wonderful MongoDB, a very powerful document-oriented database. Unlike more familiar relational databases, which store data as linked records with fields of uniform size and type, MongoDB stores collections of documents of arbitrary size and type. The documents we store are your log entries, which can be anything you want to collect and analyze --and whose schema may change at any time.

What about data security?

In an application like this, it's essential to authenticate securely and with certainty the agent sending in logging information or making a query; otherwise, anybody could read from or write to your data. We use the industry-standard OAuth protocol for authentication: our servers only accept requests (either reads or writes) that are correctly signed with pairs of keys that we provide on a per application basis. The default logworm client tools automatically do the signing of requests for you, as do the Ruby wrappers that we provide in our gem.

Please be advised that at least in this initial version, the communication between clients and server is transmitted in the open (ie, via HTTP and not via HTTPS). This means that an intruder could, in theory, sniff the packets and see the data that goes back and forth – including individual log entries, queries, and query results. We will start offering HTTPS support in the near future, with the corresponding cost and performance implications

How much does it cost?

We will always provide a free plan that allows you to store and analyze a limited number of entries in a limited number of log tables. While in Beta, the free plan includes 3 tables of up to 10MB each, which is enough for roughly 20,000 web request entries. The entries in each tables will "expire" when the table becomes full, with the oldest entries being removed first. (These are known as "capped collections". See this page about MongoDB's capped collection for details.).

We've not yet worked out the details for advanced accounts, but pricing will depend on two factors:

• number of log tables
• number of log entries

We will also offer a backup plan, in case you want to keep an archive of expired log entries.

Can I get my raw data out?

Absolutely. Unlike Google Analytics, which doesn't allow you to retrieve your raw data, you will always be able to retrieve your logworm data – quite naturally, since you own it. We will always let you take your data, port it to a different service, or just download it and analyze it yourself.

To be honest, we haven't yet finished the coding of our export functionality yet, but we'll make it the top priority soon – and as soon as you request it.

Back