Log Rate Notifications - Papertrail Blog

The revolution will be verbosely {,b}logged

Log Rate Notifications

Posted by Larry Marburger on

Log rate spikes are common and often go unnoticed. They could be an indication that something went terribly wrong or a high-traffic system was
unintentionally configured with verbose logging. As of today, Papertrail can
notify you
if your log rate is higher than expected.

How is this useful?

Here are a few real situations Papertrail customers have experienced which we realized could be solved by this single, simple notification.

  • Drastic changes in usage patterns: A database went offline and all
    connected clients generated error messages.
  • Middle-of-the-night jobs: An unattended background job unintentionally logged dozens of messages. Thousands of these jobs ran every night, and it went unnoticed for months because no one happened to look at the logs while it was running.
  • Forgotten verbose logging: One customer was troubleshooting a firewall problem and enabled iptables logging of every packet. Great idea, but it didn’t get disabled until they found it a few hours later.

In any of these situations, it would have been helpful to know at the time the change occurred instead of being surprised much later.

Putting it into practice

Set a log rate on your account and Papertrail will notify everyone
interested in usage notifications if it’s been exceeded for at least 10
minutes. System reboots or other momentary spikes won’t trigger this
notification. We’ll send an email at most every 6 hours linking directly to
the logs at that point in time so you can see what was happening.

Papertrail is in a unique position to call attention to things that are
clearly unusual. It’s our goal to give you better visibility into your logs
and usage.