The revolution will be verbosely {,b}logged

Customize receipts with VAT ID, address or other text

Posted by @troyd on

You may have already seen Papertrail’s PDF receipts, which are shown on the Purchases tab within Account.

You’ll see a new section on that page: “Customize Receipts.” Perhaps a tax authority requires a VAT identification number be on every receipt, or an accounting department requires a mailing address or vendor code.

Enter anything you want in this form, one line or multiple, and Papertrail will include it in the header of PDF receipts. While Papertrail can’t protect you from tax authorities or accounting departments, we can sure make it easier to satisfy them! Enjoy.

Constrain searches with attributes (like severity)

Posted by @troyd on

As of today, Papertrail has a new search operator: an attribute. Use an attribute-specific keyword to examine only a specific part of a log message.

Here’s a live example returning messages with the severities emergency, critical, or error.


This search will search only the program/tag element of log messages, rather than the whole message.


These attributes are available: severity, facility, sender, program, and message.

You’re probably already familiar with AND (Papertrail’s default), OR, negation (-), and parenthesis. Attribute constraints can be used in the same searches as other constraints, like this:

"something bad" program:ssh -noise

Why are these useful?

The program, sender, and message attributes permit more granular searches against data which Papertrail already examined. The severity and facility attributes allow new portions of log messages to be examined.

We added these to:

  • Build more powerful searches. program:(apache OR sshd) severity:crit is useful on its face.

  • Incorporate fields which weren’t previously searchable. Until today, the facility and severity were only included in archive exports and API responses. They’re now searchable.

  • Permit faster searches. Papertrail is subject to the laws of physics, as much as we try to optimize around them. When you know what you want and can codify it, Papertrail can examine less data and return results faster.

  • Eliminate false positives. A search for sshd finds all logs containing the string sshd anywhere, whether the message was sent from the sshd program or the body simply contained sshd. Use attributes to search for either specific case.

Play it out

In designing the grammar for attributes, we tried to maintain our starting point that “it just works.” A user would rightly expect this search to work, and it does:

("something bad" program:ssh -noise)
OR severity:error

We took that further: values can be nested to reduce typing, and without reducing the readability of the search. For example:

("something bad" program:ssh -noise)
OR program:(raid5tools ethtool)

Negation (-) works uniformly:

-("something bad" program:ssh)
OR -program:(raid5tools ethtool)
OR -noise

Grammar police

We debated how to handle multiple constraints within an attribute which can have only one. In Papertrail, the search a b means a AND b. That’s logical because AND is possible.

However, that doesn’t make sense for attributes which a log message can only have one of. program:(abc AND def) is almost never true. (The “almost” is if abc and def are different portions of a single program value, a search which would not intentionally be performed.)

4 of the 5 attributes - all except message - can only have a single value per message, so AND is never relevant. Because of this, we chose to default all attributes to OR. program:(a b) means program:(a OR b).

Saving typing

Portions of severity or facility names work fine too. For example, this works:

severity:(crit emerg)

No need to type all of critical and emergency. One argument can match multiple values, too. For example, this will match logs with any facility containing local (like local0 or local7):


All operators, and all searches, work with all Papertrail Search methods (Web, command-line, API, and inbound links). Finally, a simple query like sshd still works just like it did: Papertrail searches the message itself, sender, and program.

Finally, since senders often represent different things, the attributes host, system, and source all work as aliases for sender. Use the one(s) which make sense to you.


We hope these operators allow more granular searches. As a fringe benefit, because your search is more tightly constraining, Papertrail is able to search less data and can often return results faster.

If you try these and have ideas, please send them over. Enjoy!

Log Rate Notifications

Posted by @lmarburger on

Log rate spikes are common and often go unnoticed. They could be an indication that something went terribly wrong or a high-traffic system was unintentionally configured with verbose logging. As of today, Papertrail can notify you if your log rate is higher than expected.

How is this useful?

Here are a few real situations Papertrail customers have experienced which we realized could be solved by this single, simple notification.

  • Drastic changes in usage patterns: A database went offline and all connected clients generated error messages.
  • Middle-of-the-night jobs: An unattended background job unintentionally logged dozens of messages. Thousands of these jobs ran every night, and it went unnoticed for months because no one happened to look at the logs while it was running.
  • Forgotten verbose logging: One customer was troubleshooting a firewall problem and enabled iptables logging of every packet. Great idea, but it didn’t get disabled until they found it a few hours later.

In any of these situations, it would have been helpful to know at the time the change occurred instead of being surprised much later.

Putting it into practice

Set a log rate on your account and Papertrail will notify everyone interested in usage notifications if it’s been exceeded for at least 10 minutes. System reboots or other momentary spikes won’t trigger this notification. We’ll send an email at most every 6 hours linking directly to the logs at that point in time so you can see what was happening.

Papertrail is in a unique position to call attention to things that are clearly unusual. It’s our goal to give you better visibility into your logs and usage.

More accessible saved searches

Posted by @troyd on

We’re pleased to release an incremental improvement to make saved searches more accessible while browsing logs, as well as easier to edit. While in the log viewer, click “Searches”:


What’s new?

This screenshot reflects 2 improvements:

  • Searches are loaded with the page, rather than when “Searches” is clicked. Also, the list now expands right next to the button. While these changes might only save a second or three, changing searches feels much snappier.

  • It’s obvious how to edit any existing search, like to change its query or activate an alert. When your current query matches a saved search, the related search is highlighted.


Simpler, better automatic removal of dormant log senders

Posted by @troyd on

After a log sender permanently stops generating logs (like because a system was deprovisioned), Papertrail can automatically remove it for you.

Until today, Papertrail automatically decided whether or not to remove senders based on log destination and sender settings. We’ve made the behavior more configurable and more explicit. We hope this eliminates what might have been a manual (or simply overlooked) process.

How it worked

Papertrail relied on a basic heuristic to infer whether long-idle senders (which no longer represent any active logs) could be removed. The heuristic would only delete senders which had not been manually modified, and as a result if the sender began logging again, the new automatically-created sender would match the deleted one.

This isn’t enough information to reach the right decision, though, because 2 environments with otherwise-identical settings might prefer different behavior. Modifications aren’t a reliable indicator of preference.

Also, the heuristic was generally too conservative, so inactive senders would linger until they were manually removed. If you’ve ever thought “Hey, my Papertrail Dashboard seems to be accumulating old senders. I should remove them,” this was why.

Finally, as a byproduct of not being explicitly configurable, its behavior wasn’t obvious enough.

What changed?

We removed the magic and replaced it with a simple yes-or-no choice.

Product design means balancing automatic behavior against explicit choice. In this case, savvy people have different preferences, the heuristic can’t fully consider them, and an explicit decision isn’t hard to make. This problem is better solved by a “dumb” yet explicit implementation.

There’s now a simple “Yes, expire” or “No, don’t” option for each log destination. When enabled, idle senders are removed 2 days after their most recent log message is no longer searchable, but never less than 1 week after they’ve quit sending.


We needed to gracefully transition existing senders, so that no temporarily-idle systems are removed yet no one needs to manually remove hundreds of senders.

To that end, as part of this transition, senders with no searchable logs which haven’t been active for 1 month will be removed. 1 month was chosen as a very conservative threshold of “inactive.” Remember that no logs are being removed; the sender is simply a container for a tiny bit of metadata, and adding it again is simple.

Senders created after today will honor the policy set for its log destination. By default, it is enabled, and for most environments, auto-deletion is appropriate. To prevent idle senders from being deleted, change the log destination setting. Existing senders can also be changed via API.

If we can remove a large number of existing senders for you, please ask.

Live now

We’ve tested this with a few customers and it’s made their Papertrail Dashboard a more accurate view of their environment with no effort. In this case, a simple decision does means less work.

As usual, please send any comments or suggestions. Enjoy!

Welcoming Larry Marburger to Papertrail

Posted by @troyd on

After more than 2 years of interacting with Larry Marburger as a Papertrail customer, technologist, entrepreneur, and trusted friend, I’m incredibly pleased to welcome him to Papertrail.

Many users know and use a service which Larry helped create, CloudApp (, for capturing screenshots and sharing files. In fact, we used CloudApp at Papertrail long before I knew Larry personally.

Opportunities often arrive in unusual forms and our job is simply to try not to get in the way. Such was the case here. When CloudApp became a Papertrail customer, I said hello, and in the 2 years since, Larry has been a sounding board for Papertrail’s product ideas. Helping make these ideas a reality was the logical next step.

His in-the-trenches experience with API design, Ruby, metrics, and performance optimizations, coupled with his experience as an entrepreneur and a customer, let Larry see a unique end-to-end view: what’s truly useful, how it should work, and how it should be implemented.

Larry resides in Harrisburg, PA with his wife and two kids. In his spare time… wait, he has two kids. Larry does find time to help maintain Ruby Bundler, listen to podcasts, and partake in the official Papertrail morning beverage. You may see him at Folklore Coffee in Elizabethtown, PA.

We’ve been fortunate to work with Larry as a customer and advisor and we’re even more fortunate to welcome him to Papertrail as a contributor.

Free, un-copyrighted NDA and employment contracts

Posted by @troyd on

Papertrail’s staff grew in 2013, and as part of hiring employees, the company obtained employment agreements. These are fairly standard contracts that all technology companies use. If you’ve worked in the technology field, you’ve probably signed one.

We wanted to go a bit further, though. We got tired of not having easy-to-understand contracts and we want to start collaborating on something better. All stakeholders should be able to easily review, see changes to, discuss, and hopefully, edit and improve them.

Our stakeholders use git and Markdown. That’s why Papertrail’s employment agreement and NDA contract templates are now available on GitHub. They’re free and completely unencumbered under the Creative Commons Zero license.

Important: as the README states, Papertrail does not warrant that these are suitable for any purpose. These do not replace an attorney. Any use of these contracts is at your own risk and should be under the guidance of your own legal advisor.

Our goals:

  • make each party’s expectations obvious
  • improve the contracts over time. Not only are these contracts not set in stone, we fully expect them to change. Idea? Submit a pull request or issue.
  • provide other companies with a head start on, and chance to see the topics and changes which come up in, these agreements.

Fork, edit, and enjoy!

Self-service protocol options

Posted by @lmarburger on

If you’ve ever needed to log from systems or services which don’t support TLS encryption, check out the new log destination options available today.

Every Papertrail log destination has always supported sending logs over TCP with TLS encryption or UDP without encryption. Some systems and services can’t use TLS encryption and can only send logs in plain text TCP. This required manual intervention on our part to flip the switch to accept these logs. More people have been requesting this change so we added it as a self-service option giving you control over how we listen for logs from your systems.

Adding a new log destination or editing an existing one will present new options to enable or disable any of the supported syslog protocols: TCP with TLS encryption, plain text TCP, and plain text UDP. All options use the same port number regardless of the protocol and encryption used.


Here are a few scenarios where these new options will be of use to you.

  • Security: Enable only the TCP/TLS option and we won’t listen for logs sent over plain text.1

  • Fastly: Logs drained from Fastly can only be sent over plain text TCP. After creating a new destination to use with Fastly, edit it, check the TCP: Plain text option, and configure your Fastly account to log to this destination.

  • Multiple Senders: Several systems using the same log destination can be configured to log using the protocol and encryption that makes sense for that system. Use a single destination for all of your systems.

  1. Because UDP is connectionless, it’s possible for a system to send unencrypted logs over the network even if Papertrail isn’t configured to accept logs over UDP. Disabling UDP on a destination will drop those those logs and they won’t make it into Papertrail so you’ll know immediately that something isn’t configured correctly. 

Welcoming Leon Sodhi to Papertrail

Posted by @troyd on

We’re thrilled to officially welcome Leon Sodhi to Papertrail. Leon started in May, 2013 as Papertrail’s support engineer, so you may already have met him. Leon’s thoughtful disposition, attention to detail, and readiness to dive in and reproduce a problem have saved time for hundreds of customers.

Whether he’s helping identify an rsyslog configuration problem, patching NLog to properly handle multi-line events, discussing the pros and cons of encrypting logs with TLS, or using tcpdump to make quick work of unusual traffic, Leon is improving someone’s day.

Leon signs in from Ashford, Kent, England.

In his spare time, he recently built an electronic die. Hold a button down and a “die” (single dice) is rolled. The LEDs on the right show the result, like this.

He’s well on his way to running Linux on an 8-bit microcontroller, not unlike the original Nintendo.

We couldn’t be more thrilled or more fortunate to have Leon helping Papertrail users every day. If you come by Papertrail’s chat room, please introduce yourself and say hello.

Choose which email(s) you and others receive

Posted by @troyd on

As of today, Papertrail offers new ways to control which emails should go to which team members, such as to only send account usage updates to an operations team lead and credit card receipts to an accounting role account.

Team-wide controls are within Account. Your own preferences are duplicated on your Profile.

Team-wide email options

Here’s how it works.


Papertrail sends two types of emails:

  • Usage. Examples: 90% of log data transfer; pay-as-you-go additional usage updates.

  • Billing. Examples: payment receipt; charge failure.

The specific emails and recipient controls depend on how you consume Papertrail’s service. For example, if Papertrail is not charging your credit card, billing email controls aren’t relevant and thus aren’t available.

Previously, usage notifications were sent to all users with full access to your Papertrail account. Billing notifications were sent to the first accountholder.

Common situations

  • Only send usage notifications to a few team members, not everyone.

    Within Account, visit Members. Enable or disable recipients.

  • Also send billing notifications to an accounting@ alias (which does not have a Papertrail account).

    Within Account, look for the “Purchases” tab. Add the address. Optionally, set preferences for Papertrail accountholders in Members.

  • I want to change which emails I personally receive.

    Head over to your Papertrail Profile. This is an easy way to edit only your own options.

Personal email options

Questions? Ideas?

Let us know, and enjoy!

Smarter sorting in the Dashboard finder

Posted by @troyd on

Papertrail’s Dashboard finder now shows more specific matches first. Type any saved search, system, or group name. The most specific matches will reliably be the first results.

What changed?


Land on the Dashboard, press s (or click in the text box near the top of the page), and type the name of a group, saved search, or system. Matching saved searches, groups, and systems are shown. For searches, the finder also considers the name of a group that a saved search is associated with.

Until today, the results were sorted only by type, with saved searches first. However, saved searches were shown when the name of the related group matched, which led to a lot of matches.

As of today, Papertrail differentiates between matching resources (for example, the name of a saved search) and one-degree related resources (for example, the group that the matching search is related to).


Imagine entering Mega Service in the finder. A group named “Mega Service” should always be shown first, well ahead of its searches (like “Important errors”) and slightly ahead of similarly-named groups (like “Staging Mega Service”). Now it is:


The current behavior is also retained, which means the finder still serves as a convenient account-wide search. For example, “Mega Service” will still match all 3 resources above. They’ll be shown after any more-specific matches, so now it’s also a fast path to an exact view.

We hope this makes the Dashboard finder a faster jumping-off point for more debugging sessions. Enjoy.

An obvious, understandable live “tail” icon

Posted by @troyd on

Papertrail now has a more obvious live “tail” status icon that’s easier to understand. When the viewer is loaded, the live tail status looks like this:


Clicking the icon is one way to pause live logs, and now it communicates that. When clicked, log streaming pauses, so the icon changes to say what clicking the icon again will do:


Finally, when viewing older logs, clicking the icon is a quick way to return to the present time (and resume live logs). The icon changes to communicate that:


Where we started

Until this week, the icon always looked like this:


.. and was either lit (as shown) or dimmed.

When the arrow was first implemented, it wasn’t clickable. Later, we and others wanted to be able to quickly return to current logs and that was added. As we learned from conversations with users and our own experience:

1. The arrow gives no hint of being clickable. Some readers may be thinking “It is?!” right now..

2. When clicked, or after scrolling up to historical logs, the icon became less vibrant. This was meant to show that live tail was paused. This one also didn’t hint at its interactivity. Also, even after realizing the arrow was clickable, many users still didn’t know what it did.

Finally, the lit and dimmed icons were so subtly different that some users didn’t notice when live tail was paused.

Obvious, yet elegant

An interface element needs to be obvious enough to be discovered the first time, yet elegant - and out of the way - for every subsequent use. Although these two considerations aren’t polar opposites, they’re close.

This icon leaned heavily towards elegance: it looks good and after being used a few times, the behavior is fairly clear. If Papertrail only served a tiny set of power users - 10 people who used and talked about Papertrail all day - a subtle icon with subtle behavior might have done the job. Papertrail serves the Internet, though, and with thousands of users, it didn’t.

What we considered

In a word: everything. Papertrail uses GitHub pull requests to discuss product changes, including this one. GitHub posted a giant screenshot of one of their long-lived pull requests and our pull request for this icon change isn’t too different.

Over five months, we debated - and in many cases, implemented and tried - different solutions. Here’s three that we considered and rejected:

1. Change something other than brightness


When scrolled up, a red exclamation point indicates that something important has changed: logs are no longer streaming. Problem is that it’s actually too loud, and when the exclamation point is black or gray, not obvious enough.

It’s trying to show too many things (arrow, exclamation point) at once, too, which was a sign we’re trying to shoehorn too much meaning into one icon. And other than when the tooltip is active, it’s no more obvious that the icon is clickable.

2. Add words


Okay, the default icon needs to be easier to understand understand. Let’s add some words and use an icon that the world recognizes. Putting words underneath definitely helped indicate that clicking would do something. This variation is confusing because the icon shows what clicking will do (pause), yet the words said the current state (live tail).

3. In-app tutorial

One option was implementing a tutorial for new users instead of making the icon behavior more obvious. This was easy to reject: when we use other apps, we rarely finish the tutorial (next.. next..) and almost never enjoy them. If it’s not good enough for us as users, it’s sure not good enough for our users.

Also, a standalone tutorial was already one click away (and linked to during setup), so the subset of users who might have enjoyed an intro were probably already satisfied. The icon needed to be clearer to the rest of us.


We tried variants of these, like just the icon, just words, different words (live, tail), and using the icon to show the current state instead of what will happen when clicked.

In testing, we realized that it needs to convey three situations, not two:

1. logs are live (and clicking will pause)
2. logs are paused (and clicking will resume)
3. logs are paused (and clicking will resume and return to current, as when viewing older logs)

Even aside from logs, these are all somewhat unusual. A user is unlikely to have encountered the same need in other apps, so the behavior needs to be pretty obvious.

That led to the result, which uses 3 different icons. Two are related (pause, tail) and clearly, concisely show what clicking will do:



One is visibly different (arrow) because when viewing older logs, one fact is most important: that clicking the icon will return to current logs.


Also, all 3 states now have short tooltips. We may change that if the mouseover becomes annoying, like by only showing the tooltips during someone’s first few uses.


What you see is the best combination of discoverability and elegance we could find. We hope this more obvious, clearer interface makes this element usable by more users.

See what you think. See a change that’s worth the tradeoff between discoverability and elegance? Something completely different? Let us know.

A more elegant time seek

Posted by @troyd on

We’ve significantly improved Papertrail’s time and date seeking. Times and dates are more reliably recognized, times honor your Papertrail profile time zone, and more formats are supported.

What changed?

In the event viewer, you’ll see the same clock icon. As usual, clicking it will expand the time and date seek:

While the interface served its purpose well, the seeked-to location sometimes didn’t match expectations. Here’s what we did to eliminate those cases.

Better parsing

First, as of today, the time or date you enter is much more reliably recognized. This eliminates all known cases where dates or times were mis-interpreted. Nearly any understandable date or time format should work, like:

4:06 pm
yesterday at 5:30pm
3 hours ago
2013-08-20 04:12:55
5 AM Friday

Profile time zone

Second, the time is now interpreted according to the time zone set in your Papertrail user profile. Previously, Papertrail used the time zone of the computer you are browsing from. More below on how this came together.

More formats

Finally, additional time and date formats are supported. When copying and pasting from a terminal window or customer support email, almost any existing time representation should now be recognized. When you know when something happened, start there.

Under the hood

Implementing this elegantly was more challenging than we expected. To us, “elegant” meant:

  • honoring the Papertrail time zone setting, since log timestamps are displayed in the profile zone. Seeking to 9 AM (according to the workstation time zone) and arriving at messages saying 5 PM (in the profile zone) wasn’t intuitive at all.
  • displaying the time instantly as I type each character. We tested server-based parsing too, but even when implemented as a standalone Web service, it doesn’t feel as responsive.
  • supporting relative times. I want to type what’s on my mind, even when it’s as simple as 20 minutes ago.
  • recognizing multiple time formats, so that I don’t need to check the docs and reformat my timestamp
  • treating input as past times (as Papertrail already did), so 3 PM means “the most recent 3 PM to have already occurred” rather than “today at 3 PM”

Most other time and date interfaces we’ve seen omit at least one, and often two or three, of these conveniences. After implementing them for Papertrail, we understand why.

Biasing times towards the past was particularly challenging, since 3 AM yesterday in another time zone may be a completely different day. The time zone adjustment must occur prior to interpreting the time. The intersection of these 5 features leads to Sugar.js. We appreciate the help of its primary author, Andrew Plummer.

How is it?

Our aim was to make time seeking predictable, responsive, and complete – in a word, elegant. If you experience anything else, we want to know. Enjoy!

Search Papertrail logs right from your browser

Posted by @troyd on

Papertrail now provides an easy way to start searching logs right from your browser, without changing your search engine. Here’s Chrome waiting for a log search query:


How to get it

The next time you visit Papertrail’s Dashboard or any other page, the browser will detect Papertrail’s site search and make it available (without affecting any existing search).

This works because Papertrail exposes a standard search interface (OpenSearch) which Chrome and most other browsers detect. You’ll be prompted to login if you aren’t already logged in.

How to search

Chrome: Type the first few characters of “” or “Papertrail” in the URL bar. When autocompletion starts to suggest Papertrail, hit tab. The URL bar will change to “Search Papertrail” (see above). Your input will be sent to Papertrail as a search query.

Firefox: Browse to Papertrail’s Dashboard and, while on that page, click the icon for your current search provider. A menu will appear with Add “Papertrail.” Select it and Papertrail will be an option for future searches:


Safari: Install OpenSearchForSafari.

Mac users should also check out Papertrail’s OS X widget.


At-a-glance sender log data transfer visibility

Posted by @troyd on

We just made identifying high-volume senders easier. Now when you view information about a group, you’ll see a pie chart of currently-searchable log messages broken down by sender.

Here’s an example (credit: @lmarburger):


In this example, 81% of currently-searchable log messages were generated by a sender called engine.

When to use it

When one sender generates a flood of logs, or you’re curious which senders are unusually verbose, this is the place to start.

Combine with Papertrail’s free log filtering, verbosity tips, and minimum alert thresholds to establish and act on the log velocity that’s best for you. We’ll continue to add ways to see and control velocity.

How to use it

To see a chart, visit the Dashboard and click the name of any group, such as “All Systems”:


Here’s the chart code (JavaScript). Thanks to Stephen Boak for creating the pie chart.