Fellow Seattleite Ethan Schoonover created a color scheme called Solarized, and as of today, Papertrail uses its palette when colorizing log messages. Solarized logs look like this:

Solarized is “a sixteen color palette designed for use with terminal and GUI applications.” Solarized is uniquely well suited to Papertrail’s needs because its readability comes mostly from differences in hue (rather than from excessive use of contrast) and because the palette works well on different backgrounds.

As you can see, the result is more readable color text and less eye fatigue. That it’s also better looking is a nice bonus.

This is live now and is automatically used when viewing colorized logs in Papertrail. Enjoy!

Papertrail’s event viewer now honors ANSI color codes in log messages. Here’s an example:

The light blue and purple colors were chosen by the application which generated these logs (Ruby on Rails) and are embedded in the log messages. The messages contain ANSI color codes. These escape characters can set the foreground color, background color, and treatment (such as bold) for sections of the message.

This is live now. To use it, simply change your framework or app’s the logging configuration in your Web framework or custom app logs to generate colorized logs. Colors will be shown in Papertrail automatically. 

Papertrail already automatically colorizes the orange sender name and blue program name, since they are links to show related context (see Event Viewer Intro).

Finally, a word of caution: it’s easy to go overboard here. A tiny, tiny bit of color goes a long way.

Please send any comments (or screenshots!) and enjoy.

In the AWS Ruby blog, Trevor Rowe covers Logging HTTP Wire Traces with the AWS SDK for Ruby (aws-sdk gem) and overriding the default Logger class.

By default, aws-sdk logs to $stdout for Ruby apps or Rails.logger for Rails apps. Trevor’s example passes in an alternative Logger instance that is backed by $stderr:

AWS.config(:logger => Logger.new($stderr))

Let’s send these logs to a syslog destination like Papertrail, using Papertrail’s open-source remote_syslog_logger gem.

remote_syslog_logger

remote_syslog_logger provides a Logger-compatible class that’s backed by a UDP remote syslog destination (rather than an instance of IO, such as $stderr). To use it, add remote_syslog_logger to your Gemfile (or require it), then instantiate RemoteSyslogLogger with the syslog destination hostname and optionally, port. For example:

AWS.config(:logger => RemoteSyslogLogger.new('syslog.domain.com', 1234)

See Papertrail’s Add Systems page for these settings.

This works without any other system-wide config changes. A RemoteSyslogLogger instance can be passed to any Ruby class which expects a Logger instance.

Customize

By default, RemoteSyslogLogger will use its running program as the program name in the syslog message. Often this is just rails. Let’s set the program to something more descriptive, like myapp-development. Let’s also enable the AWS Ruby SDK http_wire_trace option to output HTTP headers (which one would typically only do in development). Here it is:

AWS.config(:logger => RemoteSyslogLogger.new('syslog.domain.com', 1234, :program => "myapp-#{RAILS_ENV}"), :http_wire_trace => true)

The README has more.

Extend

To use two Loggers, one could wrap RemoteSyslogLogger within a container class which simply invoked each of them.

Other ways to accomplish a similar result include writing logs to local syslog and configuring it to transmit them, or write to a text log file and use a daemon to watch and send it (see rsyslog, syslog-ng, or remote_syslog).

Happy debugging!

Papertrail’s log viewer now supports switching groups. We’re always looking for small tweaks that make a big impact on our own, and hopefully, your workflow. Thanks to Ben Fyvie for suggesting this.

Here’s how it works. In the event viewer, the upper left corner has changed to show the current group or sender:

Click it to see all groups:

Choose the one you want to change to. The active search and seeked-to time are retained.

Here’s how this tiny change helped me this morning. I was troubleshooting a problem by looking at Papertrail’s own Web servers group. I’d already seeked to a time and then searched to find the HTTP logs I was interested in. They led me squarely to a problem with logs from another part of our stack.

Instead of reloading the event viewer on another group and going through that again, I just changed to the group I wanted. Papertrail retained my search and time, so the log messages that I cared about were already highlighted. I was able to continue troubleshooting without losing my train of thought.

This is common in my troubleshooting process, whether it’s changing from production logs to staging ones (to see whether the same error occurred there), or from the server logs of an API request to the client’s logs. Now context stays with me.

Also, when I need to see both side-by-side, the dropdown lets me open a different group in a new browser tab. Finally, my saved searches are more useful now that they’re easy to use with any group.

Thanks to Ben for this idea. Is there a small tweak that would improve your workflow? We want to know. Enjoy!

We’ve just added support to graph the number of log messages matching a search in StatHat.

StatHat provides a simple way to graph important metrics and easily measure how it is performing compared to the past. It provides ways to receive alerts based on changes in data or absolute thresholds and automatically lets you know about important changes to the metrics you send.

To create a search alert, select StatHat from the list of search alerts and fill out the details:

Once you’ve done that, just sit back and wait for matching messages to arrive and a graph to be created for you. Soon you’ll have a graph that looks like this:

Special thanks to ejholmes on GitHub for the initial work on this search alert integration and for StatHat for adding a JSON content type for their API to make it faster and more efficient for us to send the stats.

To learn how we implemented this alert, check out the source code on GitHub or ask me directly in our chat room.

— Eric

When we started Papertrail, it was within a company called Seven Scale (which we’d used for prior ventures). As Papertrail has grown, it no longer made sense to use the Seven Scale name.

Over the next few weeks, references to “Seven Scale” will change to “Papertrail Inc.” Our customers will see zero impact from this change.

Thank you for your support and happy logging.

We’re proud to announce the Dashboard Finder, which is now live on Papertrail’s Dashboard. Try it:

How it came to be

The Dashboard is where you’ll land after logging in to Papertrail. It shows groups, saved searches, and a subset of log senders. Until today it had two weaknesses:

1. Navigating to resources not shown on the Dashboard took at least a second step. This was common with more than a few senders, when only a few were directly reachable from the Dashboard:
2. As more groups and saved searches were created, finding the best one required knowing what you want and where it is. For example, two groups and five searches could relate to the “Acme Anvil” product, and I may have only created one of them myself: 

Until today, the Dashboard needed a quick way to get anywhere and an easy way to see everything.

Announcing the Dashboard Finder

Over the last few months, we’ve been trying different ways to fill these gaps, using them ourselves, and iterating. Today we’re proud to unveil the Dashboard Finder:

Small in screen real estate, the Finder hides its power. Have an idea where you’re going? When you arrive at the Dashboard, hit the “s” key (or click in the text box) and type the name or a fragment of a saved search, group, or sender.

That’s it. The Dashboard Finder stays out of the way until you need it, then makes it easy to load any view in the event viewer.

The arrow keys work naturally. I can see and reach everything related to “Production” with a few keystrokes:

My fingers never leave the keyboard.

We hope the Dashboard Finder saves as much time for you as has for us. Like everything at Papertrail, we’re learning as we go, and we want your feedback. If you use the Finder in a way that it’s not perfect for, tell us. And if it works beautifully for you, tell us that too.

Enjoy!

As users know, Papertrail has offered painless log search alerts for some time. Notify a Campfire chat room, email a team, or kick off a PagerDuty escalation process when something important happens. Get periodic summaries about less-critical issues.

Notifications are inherently binary, though: either an alert fires or it doesn’t. You want to know or you don’t. There’s often a gray area in monitoring, though, where two of something is fine, 20 is not.

Papertrail now lets you decide that gray area. Papertrail search alerts can now include a minimum number of events that must occur (during the alert interval) for the alert to be invoked. The default and existing behavior is a minimum of 1.

What’s possible with a minimum threshold?

• Allow for endemic problems. With some issues, a constant undercurrent occurs all the time. Internet-facing problems like 404s, brute-force attacks, and site scrapers are the obvious examples. I don’t care that they happen, but I do care when they cross above what I consider noise.

• Allow for known issues. If you’re short on RAM, the fact that Linux’s “oom_killer” killed a process once to save RAM probably isn’t severe. If it kills 5, it is. This is particularly handy with code exceptions (known bugs) and services where log output can’t easily be modified.

• Detect when less-critical issues become trends. For example, I’d like to know when more than 10 slow queries occur across all of my database servers. That’s not a slow query, that’s a trend.

• Combine these. Take velocity into account when deciding what the alert should do. When 50 of something happen in a 10 minute period, it’s urgent; tell me in HipChat. Separately, email a summary to my team every day, regardless of volume.

How to use it

Use this in combination with log filtering and “Show related logs” links to ignore the events that don’t matter, react to those that appear to be important, and link directly from your admin dashboard to those that you know are relevant.

Perhaps the best part of Papertrail’s search alert threshold is that it comes without a ton more time or effort on your behalf. There’s not a zillion controls because there doesn’t need to be; there’s the one that matters.

As product designers, we always try find the knobs that solve the most problems - expose the most power - with the least, and least visible, complexity. There’s not many better examples of it than this.

Last week we quietly released a new papertrailapp.com public home page.

Our goal was simple: incorporate what we’ve learned from operating Papertrail. We’ve actually seen how people interact with the site, which wasn’t the case when the original site was designed.

While the text hasn’t changed dramatically, the new site’s layout highlights the elements that we’ve seen firsthand are important to visitors. It also does a much better job of demonstrating that we care about stylized design. It feels more modern, which is a more accurate representation of Papertrail’s product and our view of the world.

The result speaks for itself. Feel free to send comments.

Papertrail can now filter log messages on behalf of your apps and systems.

Constantly logging a noisy system error that you’re well aware of? Don’t care about requests from monitoring probes? Give us a regular expression and we’ll ignore them. No need to touch your app or system config.

Keep reading or dive right in the docs.

Why is this important?

Usefulness is in the eye of the beholder. It’s not our job to tell you what log messages might be useful, and we’d prefer that it isn’t left to your application or system’s default configuration, either. The logs that the author chose to generate may not have any relationship to the ones you care about.

Our job is to expose intelligently-chosen control and configurability. More succinctly: the knobs that matter. This is one of them.

Logging configs are harder to change than they could be, especially with multiple senders and deployment environments. Papertrail’s log filtering transfers that control from the daemon or framework author to you.

That’s true for managed services too, where a hosting provider may not be able to modify their logging policies to suit a single customer.

Also, Papertrail is in a unique position to provide infrastructure-wide logging control. In addition to message content, Papertrail can filter by sender, program, or log destination, which means there’s a single place to silence almost any unnecessary logs.

We hope this saves time which might otherwise be spent fighting with app configs.

Am I charged for filtered messages?

No. Log filtering is included with all Papertrail accounts and filtered messages don’t consume log data transfer.

Let’s see it in action

A standard Ruby on Rails app generates these messages for every page fragment (“partial”) used in a page:

Rendered layouts/_head.html.erb

Although we have easy ways to make Rails less verbose, assume I’m mostly happy with the stock logging except these messages. Knowing the list of page elements that were rendered usually serves no purpose in production, so they’re basically noise. And changing the log output either means omitting other messages (by decreasing verbosity) or monkey-patching Rails.

So, let’s have Papertrail ignore them. Use the filter:

Rendered 

In your log destination configuration (Account), it looks like this:

Users of Papertrail via hosting services may see a slightly different presentation for clarity.

Next, let’s just ignore these messages from senders that begin with “www” (like www1 and www2), not all senders. Also, we don’t care about static asset requests (like images and CSS files), either, so let’s also ignore messages from those senders that contain “/assets”. Use the filter:

www\d+ \W: (Rendered|\/assets)

See it in Rubular. The filter is a regular expression. This regex matches any sender name with “www” followed by one or more digits, then any app name from those senders, then either of the messages we want to filter.

Use it

Use Papertrail’s log filtering to customize where an app should be verbose and where it should be quiet. No need to be limited by the sending service’s default settings or the implementor’s preferences. Just decide what you don’t want and create a regular expression to match it.

Head over to Log Filtering and read more, then configure it in Account settings.

Papertrail provides a Heroku add-on for aggregating and managing logs from apps, including dyno/app output, router requests, and platform events. This calls for a consistent interface between the Heroku Dashboard and Papertrail (when accessed as an add-on), so Papertrail displays part of the heroku.com site header.

Until recently, the header was, well, really tall:

We’re all about making the most of every pixel on any Papertrail interface. These pixels were not well spent. To that end, we’ve been exchanging ideas with Heroku about how to make the header shorter.

On Wednesday, Heroku’s Zeke Sikelianos released boomerang, which is “a JavaScript widget that gives users a way back to Heroku from Addon provider sites.” The header now looks like this:

Zeke shrunk the header from 148 to 30 pixels. That 118 pixel savings is about 10% of the screen real estate on many displays, so it made a dramatic difference.

He took it further and made the header more useful by adding links to the app settings that a visitor might need. Now Web-centric add-ons feel more like an integrated experience within Heroku’s Dashboard.

We released this on Thursday and have been basking in its glory for the last day. Enjoy!

Aside from the new group-wide log context link, we made one much smaller change: saved searches are now alphabetized in Papertrail’s Dashboard.

Back Story

Until today, saved searches were shown in the order that they had been created. This made sense with a relatively small number of searches (and when a relatively small set of people was creating them); at that point, when it was created was meaningful.

As more saved searches are created (and in many Papertrail groups), name starts to matter more than chronology. Also, saved searches can act as an informal institutional knowledge transfer: a database administrator might create a search that a developer finds useful, and vice versa. In that case, the search’s name - like “503 errors” or “User-facing slow queries” - is all that matters, and that’s how coworkers will discover and use it.

Enjoy, and send any comments our way or to support@.

As you’ve probably seen, Papertrail’s log viewer shows links to display a specific message in context. Click the name of a log sender or a program to jump to that message within the related messages generated right before and right after it:

What Changed?

We made a subtle but important change to the program name link, shown in blue (like “access_log” and “sshd”).

Until today, clicking the name of a program which generated a message would show that message surrounded by logs from that program on only that log sender. For example, only messages from “sshd” running on the system named “taco.”

As of today, clicking a program shows that message surrounded by all messages from that program, from all log senders that you’re looking at (typically a group). In the past, it only included the specific sender which generated the message.

The log sender link (shown in orange, like “taco”) has not changed. It still shows a message surrounded by all messages from that log sender.

Back Story

This came about based on how we saw Papertrail used by customers and from using it ourselves. In our own case, we often wanted to see a message in context with related messages from across a multi-server fleet.

We found that it was more useful than honing in on a specific system+app (which the orange link accomplishes well). One of Papertrail’s goals is to be users first, so we’re always looking for tiny improvements that improve workflow.

What It’s For

If you haven’t tried contextual links, here’s couple examples how they’re useful to us:

• History: after seeking to a HTTP request that served a user-facing error, click the program name (“access_log”) to show the preceding and following requests. With the change above, the context includes requests made to other systems in the group, like on a load-balanced Web server cluster.
• Long events: when a search matches one line in a multi-line traceback or exception, clicking the name of the log sender (“taco”) will show the entire traceback.
• Impact: spot a kernel problem in a syslog message? Click the log sender to show that message amidst messages from just that system and see the impact on apps.

Enjoy, and please send comments or suggestions.

I’m pleased to announce that Papertrail now supports a read-only account access level. Need to share short-term log visibility with a consultant? Want to spread ops visibility throughout your company, without the risk of someone accidentally changing a setting? This is for you.

Read-only users have permission to tail and search logs, view groups and saved searches, and download archives. They can’t edit, change, or delete account assets (like systems, searches, or log messages). Also, they can’t view or change payment information and are not emailed about plan limits.

Account administrators can set privileges for all other users on the Members page:

Give it a whirl and send us any suggestions.