Posts tagged devops

Papertrail’s log viewer now supports switching groups. We’re always looking for small tweaks that make a big impact on our own, and hopefully, your workflow. Thanks to Ben Fyvie for suggesting this.

Here’s how it works. In the event viewer, the upper left corner has changed to show the current group or sender:

Click it to see all groups:

Choose the one you want to change to. The active search and seeked-to time are retained.

Here’s how this tiny change helped me this morning. I was troubleshooting a problem by looking at Papertrail’s own Web servers group. I’d already seeked to a time and then searched to find the HTTP logs I was interested in. They led me squarely to a problem with logs from another part of our stack.

Instead of reloading the event viewer on another group and going through that again, I just changed to the group I wanted. Papertrail retained my search and time, so the log messages that I cared about were already highlighted. I was able to continue troubleshooting without losing my train of thought.

This is common in my troubleshooting process, whether it’s changing from production logs to staging ones (to see whether the same error occurred there), or from the server logs of an API request to the client’s logs. Now context stays with me.

Also, when I need to see both side-by-side, the dropdown lets me open a different group in a new browser tab. Finally, my saved searches are more useful now that they’re easy to use with any group.

Thanks to Ben for this idea. Is there a small tweak that would improve your workflow? We want to know. Enjoy!

We’re proud to announce the Dashboard Finder, which is now live on Papertrail’s Dashboard. Try it:

How it came to be

The Dashboard is where you’ll land after logging in to Papertrail. It shows groups, saved searches, and a subset of log senders. Until today it had two weaknesses:

1. Navigating to resources not shown on the Dashboard took at least a second step. This was common with more than a few senders, when only a few were directly reachable from the Dashboard:
2. As more groups and saved searches were created, finding the best one required knowing what you want and where it is. For example, two groups and five searches could relate to the “Acme Anvil” product, and I may have only created one of them myself: 

Until today, the Dashboard needed a quick way to get anywhere and an easy way to see everything.

Announcing the Dashboard Finder

Over the last few months, we’ve been trying different ways to fill these gaps, using them ourselves, and iterating. Today we’re proud to unveil the Dashboard Finder:

Small in screen real estate, the Finder hides its power. Have an idea where you’re going? When you arrive at the Dashboard, hit the “s” key (or click in the text box) and type the name or a fragment of a saved search, group, or sender.

That’s it. The Dashboard Finder stays out of the way until you need it, then makes it easy to load any view in the event viewer.

The arrow keys work naturally. I can see and reach everything related to “Production” with a few keystrokes:

My fingers never leave the keyboard.

We hope the Dashboard Finder saves as much time for you as has for us. Like everything at Papertrail, we’re learning as we go, and we want your feedback. If you use the Finder in a way that it’s not perfect for, tell us. And if it works beautifully for you, tell us that too.

Enjoy!

As users know, Papertrail has offered painless log search alerts for some time. Notify a Campfire chat room, email a team, or kick off a PagerDuty escalation process when something important happens. Get periodic summaries about less-critical issues.

Notifications are inherently binary, though: either an alert fires or it doesn’t. You want to know or you don’t. There’s often a gray area in monitoring, though, where two of something is fine, 20 is not.

Papertrail now lets you decide that gray area. Papertrail search alerts can now include a minimum number of events that must occur (during the alert interval) for the alert to be invoked. The default and existing behavior is a minimum of 1.

What’s possible with a minimum threshold?

• Allow for endemic problems. With some issues, a constant undercurrent occurs all the time. Internet-facing problems like 404s, brute-force attacks, and site scrapers are the obvious examples. I don’t care that they happen, but I do care when they cross above what I consider noise.

• Allow for known issues. If you’re short on RAM, the fact that Linux’s “oom_killer” killed a process once to save RAM probably isn’t severe. If it kills 5, it is. This is particularly handy with code exceptions (known bugs) and services where log output can’t easily be modified.

• Detect when less-critical issues become trends. For example, I’d like to know when more than 10 slow queries occur across all of my database servers. That’s not a slow query, that’s a trend.

• Combine these. Take velocity into account when deciding what the alert should do. When 50 of something happen in a 10 minute period, it’s urgent; tell me in HipChat. Separately, email a summary to my team every day, regardless of volume.

How to use it

Use this in combination with log filtering and “Show related logs” links to ignore the events that don’t matter, react to those that appear to be important, and link directly from your admin dashboard to those that you know are relevant.

Perhaps the best part of Papertrail’s search alert threshold is that it comes without a ton more time or effort on your behalf. There’s not a zillion controls because there doesn’t need to be; there’s the one that matters.

As product designers, we always try find the knobs that solve the most problems - expose the most power - with the least, and least visible, complexity. There’s not many better examples of it than this.

Papertrail can now filter log messages on behalf of your apps and systems.

Constantly logging a noisy system error that you’re well aware of? Don’t care about requests from monitoring probes? Give us a regular expression and we’ll ignore them. No need to touch your app or system config.

Keep reading or dive right in the docs.

Why is this important?

Usefulness is in the eye of the beholder. It’s not our job to tell you what log messages might be useful, and we’d prefer that it isn’t left to your application or system’s default configuration, either. The logs that the author chose to generate may not have any relationship to the ones you care about.

Our job is to expose intelligently-chosen control and configurability. More succinctly: the knobs that matter. This is one of them.

Logging configs are harder to change than they could be, especially with multiple senders and deployment environments. Papertrail’s log filtering transfers that control from the daemon or framework author to you.

That’s true for managed services too, where a hosting provider may not be able to modify their logging policies to suit a single customer.

Also, Papertrail is in a unique position to provide infrastructure-wide logging control. In addition to message content, Papertrail can filter by sender, program, or log destination, which means there’s a single place to silence almost any unnecessary logs.

We hope this saves time which might otherwise be spent fighting with app configs.

Am I charged for filtered messages?

No. Log filtering is included with all Papertrail accounts and filtered messages don’t consume log data transfer.

Let’s see it in action

A standard Ruby on Rails app generates these messages for every page fragment (“partial”) used in a page:

Rendered layouts/_head.html.erb

Although we have easy ways to make Rails less verbose, assume I’m mostly happy with the stock logging except these messages. Knowing the list of page elements that were rendered usually serves no purpose in production, so they’re basically noise. And changing the log output either means omitting other messages (by decreasing verbosity) or monkey-patching Rails.

So, let’s have Papertrail ignore them. Use the filter:

Rendered 

In your log destination configuration (Account), it looks like this:

Users of Papertrail via hosting services may see a slightly different presentation for clarity.

Next, let’s just ignore these messages from senders that begin with “www” (like www1 and www2), not all senders. Also, we don’t care about static asset requests (like images and CSS files), either, so let’s also ignore messages from those senders that contain “/assets”. Use the filter:

www\d+ \W: (Rendered|\/assets)

See it in Rubular. The filter is a regular expression. This regex matches any sender name with “www” followed by one or more digits, then any app name from those senders, then either of the messages we want to filter.

Use it

Use Papertrail’s log filtering to customize where an app should be verbose and where it should be quiet. No need to be limited by the sending service’s default settings or the implementor’s preferences. Just decide what you don’t want and create a regular expression to match it.

Head over to Log Filtering and read more, then configure it in Account settings.

Almost everything we touch at Papertrail has a timestamp, which means it also has an expressed or implied time zone. Here’s more about why and how we work with time zones, then an introduction to new time zone-aware search alerts.

Who Cares?

Modern time zones came about in 1884, when a Canadian railroad engineer proposed a “universal day” based on a world meridian. At the International Meridian Conference, the Observatory of Greenwich was chosen as the meridian and zones were created every 15 degrees of longitude (proceedings).

Papertrail encounters a few issues that 19th century railroads did:

• Zones can change. The items we care about - log messages, users, servers - can all be mobile. Like trains and passengers, they can start, traverse, and end in different places.
• Reasonable people expect different things. Sysadmin Sally might have servers in 6 countries, set them to GMT, and convert to local time in her head. She probably expects something different than Developer Dan, who logs in once a week to troubleshoot a problem (and shouldn’t need to care about that conversion). Sally and Dan may be coworkers.
• Humans aren’t the only audience. Aside from operations, Papertrail needs to present the right zone and level of zone awareness to webhooks, permanent archives, and other targets.
• The status quo isn’t great. Log aggregation often transfers the burden on to the user, either by “punting” and treating timestamps as strings rather than times (so everyone has to do what Sally does) or by converting them to a single zone.

Our goal is to deliver a service that doesn’t require knowing any of that.

OK, How?

As much as possible, our starting point is the principle of least astonishment: Papertrail should just work, and then be tweakable where needed. We’ve spent effort to:

• Let servers and apps live anywhere and configured however the operator wants. Multiple data centers, cloud presences, and PaaS deployments are the norm. They often don’t - and should’t need to - use the same time zone.
• Deliver a sequential view of events across all infrastructure, without requiring reliable time sync. Troubleshooting interactions between 2 servers is very difficult when the events are out of order, and if Papertrail blindly trusted local timestamps, they often would be.
• Support distributed teams. Each engineer should be able to view messages in whatever zone is most logical to them. Same for timestamps provided by staff, like when seeking to a time.
• Present a consistent view of inconsistent logs. Some log messages contain explicit timestamps (ISO 8601), others have implicit zones (known only to the server), and some senders may generate logs in multiple zones concurrently (for example, OS, Web server, and app events). Manually tracking and normalizing these is a particularly demeaning use of humans.
• Make it easier for log consumers (search alerts, API clients) to ignore zones. Transfer the burden of localization on to Papertrail. More below.

Time Zone-Aware Log Alerts

As of March 29, 2012, search alerts can now receive event timestamps in any timestamp. Hit the Dashboard, find the saved search you have an alert for, mouseover it, and click “Edit.” Under “Manage Alerts,” choose the alert.

You’ll see a time zone dropdown. Choose the zone that Papertrail should convert the timestamps of matching messages to.

The Making Of Zone-Aware Alerts

Here’s the back story. We tried a few incarnations and found that even teams which are spread across time zones usually have a zone that the team thinks of as standard (what server clocks are set to, a particular office, UTC, etc.).

We also found that for emails, automatically changing timestamps to each recipient’s Papertrail account zone sounds ideal, but it’s actually not very intuitive because other than email, most alert methods inherently present a single time zone to many recipients. For example, a single chat room alert is seen by many viewers.

(There’s also an unwritten expectation with team-wide alerts that the whole team received the same thing. When a coworker pastes a log message in IM, converting it from their time zone to yours isn’t even on your radar.)

Also, treating email uniquely would mean that for the same issue, the same individual could receive a Campfire alert in one zone and an email in another. Besides violating the principle of least astonishment, that’s just dumb. There’s a better way: render each event’s timestamp as a link directly to that message in the Web viewer (where the time is localized to you). So Papertrail does that.

That’s why alert timestamps are set once for the alert. It helps that we’ve actually tried, and lived with, these decisions. 

I hope you enjoyed this introduction to time zones in Papertrail. As always, if something isn’t working well, doesn’t seem obvious, or simply can be better, tell us.

Papertrail supports the remote syslog protocol, so it accepts Web server access logs from rsyslog, syslog-ng, the tiny remote_syslog log file to remote syslog daemon, and other senders.

In that “other senders” category, here’s an elegant hack to have Apache transmit access logs directly to a remote syslog server, using a one-line httpd.conf change.

To transmit with the hostname “www1” and the program name “apache”, add this line:

CustomLog "|nc -u logs.papertrailapp.com 1111" "<134>%{%b %d %X}t www1 apache %h %l %u %t \"%r\"%>s %b \"%{Referer}i\" \"%{User-agent}i\""

This combines netcat, Apache’s CustomLog configuration directive, and Apache’s piped logs feature (which will even restart nc if it crashes). Apache outputs a syslog-framed message to a pipe and nc does the rest. The <134> is the syslog’s priority identifier for facility local0, severity info. That’s followed by the syslog timestamp, system name, and program name.

Everything after “apache” is format specifiers to generate the standard combined log format. The format can be customized. The CustomLog directive works globally and can be used in VirtualHost stanzas.

The reference example is:

CustomLog "|nc -u <destination hostname> <destination port>" "<134>%{%b %d %X}t <system hostname> <program name> %h %l %u %t \"%r\"%>s %b \"%{Referer}i\" \"%{aUser-agent}i\""

This would work for any daemons which can output to a pipe, don’t block on the output (or automatically restart the pipe program, as Apache does), and support a user-supplied template for message formatting. It’s also possible to CustomLog to pipe to the “logger” program (instead of netcat), like this:

CustomLog "|logger -t httpd -p local1.info"

.. and then use your existing syslog daemon to transmit those to Papertrail.

We formally announced public availability of Papertrail hosted log management yesterday. This is as formal as we get!

Read the announcement below (or on Business Wire).

Cloud-Based Logging Service Papertrail Launches

Hosted log management service helps engineers detect, resolve, and avoid infrastructure problems

SEATTLE—(BUSINESS WIRE)—Cloud-based log management service Papertrail™ announced public availability today. Papertrail delivers effortless log aggregation and management as a hosted service (SaaS).

In 30 seconds, an engineer or startup entrepreneur can obtain a searchable, scalable, always-on repository for their server and application log messages, over the Internet. With a simple one-line change, logs are sent to Papertrail as they happen.

Bringing all log messages together is the gateway to powerful operations visibility, monitoring, and analysis with Papertrail. For example:

• systems administrators can watch infrastructure events in realtime — from the Web, command-line, or REST API.
• developers diagnosing app problems can search for anything they know (like an IP address, error message, or customer name), or seek right to a specific time or server.
• support staff can view events from only the product that they support, without access to the underlying systems.
• architects can setup alerts to notify monitoring systems about important events (web hooks), and analyze months of logs in Hadoop to identify trends.

“Engineers often discover Papertrail as the easiest way to aggregate server log files, then they try it and realize that log collection is just the beginning,” said Troy Davis, co-founder and CEO of Papertrail’s operating company, Seven Scale. “Seeing operating system logs, app server requests, database queries, and router logs in one live interface is incredibly useful.”

A single Web request might generate a firewall log message, a Redis lookup, a dozen MySQL queries, a Ruby on Rails request log, a handful of static file logs from Apache, and syslog from a background task. Because Papertrail works with all common log methods, engineers can see these related events together as an end-to-end operational narrative.

Pre-launch customers already use Papertrail to manage many terabytes every month. It seamlessly scales with each customer’s needs, whether a few messages per hour or thousands per second. Systems and apps can reside on physical servers, hosting services like Amazon Elastic Compute Cloud (EC2), and app clouds like Heroku.

“DevOps engineers want productivity. It’s not just about more data or more features, but the right info at the right time,” Davis said. “Papertrail provides that, and in less than a minute.”

Demonstrations will be available at the O’Reilly Velocity Web Performance & Operations Conference.

Seven Scale operates the Papertrail hosted log management service, which brings smiles to the faces of systems engineers, developers, and entrepreneurs.

Everyone deserves great logging. Sign up and log to the cloud™ at http://papertrailapp.com/