The revolution will be verbosely {,b}logged

10 Minute Alert Frequency

Posted by @troyd on

Papertrail now offers a fourth time frequency for search alerts: every 10 minutes.

10 minutes is short enough for many operational problems, yet long enough to capture multiple occurrences of many problems. 10 minutes fits 2 common situations particularly well:

  • Time-sensitive problems which check for a certain numer of occurrences (using Papertrail’s minimum alert threshold). For example, 3 segfaults in 10 minutes.

  • Urgent problems which may continue for some time once they start. Waiting an hour is too long, yet there’s no need for a notification every minute while a problem is occurring.

To change the frequency of a search alert, visit Papertrail’s Dashboard, hover over the related search, and click “Edit”:

Search alert frequencies

Enjoy!

OpenSSL "Heartbleed" vulnerability summary

Posted by @troyd on

A vulnerability in OpenSSL called CVE-2014-0160 (nicknamed “Heartbleed”) was publicly announced on Monday, April 7. Papertrail:

  • Patched the HTTPS endpoint serving papertrailapp.com on Monday at 3:30 PM UTC-7 (see status blog).

  • Verified that our TLS-encrypted log endpoint is not vulnerable to the exploit.

  • Changed https://papertrailapp.com/ to use a new TLS certificate at 5:00 PM UTC-7. This certificate was generated by a different private key. Related internal passphrases were also changed.

  • Deployed forward secrecy As part of patching OpenSSL.

This vulnerability affects many, probably most, SSL-enabled Internet services in some form. We echo Tumblr’s recommendation, as reported in the LA Times: “take some time to change your passwords everywhere.” Be safe.

New Account Usage Overview

Posted by @lmarburger on

The new account usage page gives an overview of total log transfer over the past month and the senders with the highest volume. Find it linked from your account’s plan usage section.

Account Usage

Use these new graphs to see things about your usage that was previously unavailable. For example:

  • Top senders: Spot the senders logging the most data. Jump to a high-volume sender’s logs to look for unused messages which can be filtered.

  • Changes in log rate: Did a recent deploy affect log output? Maybe a debug message slipped through to production or an entire sender has been silent today.

  • Unexpected senders: Perhaps a daily job logs a good deal of unnecessary data. Unless someone is watching the events while the job runs, it’s possible no one would ever notice.

Along with log rate notifications, we hope you’re able to catch unnecessary logs quicker to trim noise and save money.

More consistent service intervals and 16/25 GB pricing

Posted by @troyd on

We’ve made 3 minor changes to make Papertrail’s service easier to understand and more predictable:

  1. The free plan now measures usage the same way as all other plans, which is from one day of a month (such as the 13th) to the same day in the following calendar month. Details are below.
  2. Existing plans with 16 GB/month of log data transfer are now less expensive. The new prices are more logical when compared to 8 and 25 GB/month plans. All existing customers on the 16 GB/month plans will automatically receive the lower pricing as well as individual emails about the change.
  3. New plans have been created with 25 GB of log data transfer per month. They fill an unintentional gap between 16 GB and 50 GB which previously required pay-as-you-go additional usage. As with all Papertrail plans, they are available with 1, 3, 7, 14, and 30 days of searchable retention and 1 year of archives.

In all cases, the changes are either less expensive or have essentially no impact. The old and new prices for Papertrail’s 16 GB/month plan are below. To change to the 25 GB/month plan or any other, see popular plans or build your own.

Free plan service interval

Until today, usage on Papertrail’s free service was measured in a slightly different way than usage on all other plans. Instead of using a fixed day of the month (such as the day that service began), the free plan used a sliding 30-day window.

This was intended to be more flexible, but it’s not. There’s no situation where a 30-day window is more accommodating than a fixed 1-month period. Also, after one has explained a sliding window 2 or 30 times, the novelty value wears off too.

As of today, all of Papertrail’s monthly plans work the same: usage is measured for a 1-month period beginning on a specific day of the month. Usually this is the day that you signed up or that the current service began.

This change has almost no impact. The log data transfer in a fixed 30-day measurement period is equally likely to be slightly higher as it is slightly lower (compared to a 30-day sliding window). All measurement information is shown on the Account page.

16 GB/month price change

Searchable retention Old New
1 day $90 $82
3 days $145 $103
7 days $175 $125
14 days $225 $150
30 days $260 $195

Questions

Please let us know if we can answer questions or clarify any of this. Here’s to simplicity.

Customize receipts with VAT ID, address or other text

Posted by @troyd on

You may have already seen Papertrail’s PDF receipts, which are shown on the Purchases tab within Account.

You’ll see a new section on that page: “Customize Receipts.” Perhaps a tax authority requires a VAT identification number be on every receipt, or an accounting department requires a mailing address or vendor code.

Enter anything you want in this form, one line or multiple, and Papertrail will include it in the header of PDF receipts. While Papertrail can’t protect you from tax authorities or accounting departments, we can sure make it easier to satisfy them! Enjoy.

Constrain searches with attributes (like severity)

Posted by @troyd on

As of today, Papertrail has a new search operator: an attribute. Use an attribute-specific keyword to examine only a specific part of a log message.

Here’s a live example returning messages with the severities emergency, critical, or error.

Example

This search will search only the program/tag element of log messages, rather than the whole message.

program:sshd

These attributes are available: severity, facility, sender, program, and message.

You’re probably already familiar with AND (Papertrail’s default), OR, negation (-), and parenthesis. Attribute constraints can be used in the same searches as other constraints, like this:

"something bad" program:ssh -noise

Why are these useful?

The program, sender, and message attributes permit more granular searches against data which Papertrail already examined. The severity and facility attributes allow new portions of log messages to be examined.

We added these to:

  • Build more powerful searches. program:(apache OR sshd) severity:crit is useful on its face.

  • Incorporate fields which weren’t previously searchable. Until today, the facility and severity were only included in archive exports and API responses. They’re now searchable.

  • Permit faster searches. Papertrail is subject to the laws of physics, as much as we try to optimize around them. When you know what you want and can codify it, Papertrail can examine less data and return results faster.

  • Eliminate false positives. A search for sshd finds all logs containing the string sshd anywhere, whether the message was sent from the sshd program or the body simply contained sshd. Use attributes to search for either specific case.

Play it out

In designing the grammar for attributes, we tried to maintain our starting point that “it just works.” A user would rightly expect this search to work, and it does:

("something bad" program:ssh -noise)
OR severity:error

We took that further: values can be nested to reduce typing, and without reducing the readability of the search. For example:

("something bad" program:ssh -noise)
OR program:(raid5tools ethtool)

Negation (-) works uniformly:

-("something bad" program:ssh)
OR -program:(raid5tools ethtool)
OR -noise

Grammar police

We debated how to handle multiple constraints within an attribute which can have only one. In Papertrail, the search a b means a AND b. That’s logical because AND is possible.

However, that doesn’t make sense for attributes which a log message can only have one of. program:(abc AND def) is almost never true. (The “almost” is if abc and def are different portions of a single program value, a search which would not intentionally be performed.)

4 of the 5 attributes - all except message - can only have a single value per message, so AND is never relevant. Because of this, we chose to default all attributes to OR. program:(a b) means program:(a OR b).

Saving typing

Portions of severity or facility names work fine too. For example, this works:

severity:(crit emerg)

No need to type all of critical and emergency. One argument can match multiple values, too. For example, this will match logs with any facility containing local (like local0 or local7):

facility:local

All operators, and all searches, work with all Papertrail Search methods (Web, command-line, API, and inbound links). Finally, a simple query like sshd still works just like it did: Papertrail searches the message itself, sender, and program.

Finally, since senders often represent different things, the attributes host, system, and source all work as aliases for sender. Use the one(s) which make sense to you.

Suggestions?

We hope these operators allow more granular searches. As a fringe benefit, because your search is more tightly constraining, Papertrail is able to search less data and can often return results faster.

If you try these and have ideas, please send them over. Enjoy!

Log Rate Notifications

Posted by @lmarburger on

Log rate spikes are common and often go unnoticed. They could be an indication that something went terribly wrong or a high-traffic system was unintentionally configured with verbose logging. As of today, Papertrail can notify you if your log rate is higher than expected.

How is this useful?

Here are a few real situations Papertrail customers have experienced which we realized could be solved by this single, simple notification.

  • Drastic changes in usage patterns: A database went offline and all connected clients generated error messages.
  • Middle-of-the-night jobs: An unattended background job unintentionally logged dozens of messages. Thousands of these jobs ran every night, and it went unnoticed for months because no one happened to look at the logs while it was running.
  • Forgotten verbose logging: One customer was troubleshooting a firewall problem and enabled iptables logging of every packet. Great idea, but it didn’t get disabled until they found it a few hours later.

In any of these situations, it would have been helpful to know at the time the change occurred instead of being surprised much later.

Putting it into practice

Set a log rate on your account and Papertrail will notify everyone interested in usage notifications if it’s been exceeded for at least 10 minutes. System reboots or other momentary spikes won’t trigger this notification. We’ll send an email at most every 6 hours linking directly to the logs at that point in time so you can see what was happening.

Papertrail is in a unique position to call attention to things that are clearly unusual. It’s our goal to give you better visibility into your logs and usage.

More accessible saved searches

Posted by @troyd on

We’re pleased to release an incremental improvement to make saved searches more accessible while browsing logs, as well as easier to edit. While in the log viewer, click “Searches”:

searches

What’s new?

This screenshot reflects 2 improvements:

  • Searches are loaded with the page, rather than when “Searches” is clicked. Also, the list now expands right next to the button. While these changes might only save a second or three, changing searches feels much snappier.

  • It’s obvious how to edit any existing search, like to change its query or activate an alert. When your current query matches a saved search, the related search is highlighted.

Enjoy!

Simpler, better automatic removal of dormant log senders

Posted by @troyd on

After a log sender permanently stops generating logs (like because a system was deprovisioned), Papertrail can automatically remove it for you.

Until today, Papertrail automatically decided whether or not to remove senders based on log destination and sender settings. We’ve made the behavior more configurable and more explicit. We hope this eliminates what might have been a manual (or simply overlooked) process.

How it worked

Papertrail relied on a basic heuristic to infer whether long-idle senders (which no longer represent any active logs) could be removed. The heuristic would only delete senders which had not been manually modified, and as a result if the sender began logging again, the new automatically-created sender would match the deleted one.

This isn’t enough information to reach the right decision, though, because 2 environments with otherwise-identical settings might prefer different behavior. Modifications aren’t a reliable indicator of preference.

Also, the heuristic was generally too conservative, so inactive senders would linger until they were manually removed. If you’ve ever thought “Hey, my Papertrail Dashboard seems to be accumulating old senders. I should remove them,” this was why.

Finally, as a byproduct of not being explicitly configurable, its behavior wasn’t obvious enough.

What changed?

We removed the magic and replaced it with a simple yes-or-no choice.

Product design means balancing automatic behavior against explicit choice. In this case, savvy people have different preferences, the heuristic can’t fully consider them, and an explicit decision isn’t hard to make. This problem is better solved by a “dumb” yet explicit implementation.

There’s now a simple “Yes, expire” or “No, don’t” option for each log destination. When enabled, idle senders are removed 2 days after their most recent log message is no longer searchable, but never less than 1 week after they’ve quit sending.

Transition

We needed to gracefully transition existing senders, so that no temporarily-idle systems are removed yet no one needs to manually remove hundreds of senders.

To that end, as part of this transition, senders with no searchable logs which haven’t been active for 1 month will be removed. 1 month was chosen as a very conservative threshold of “inactive.” Remember that no logs are being removed; the sender is simply a container for a tiny bit of metadata, and adding it again is simple.

Senders created after today will honor the policy set for its log destination. By default, it is enabled, and for most environments, auto-deletion is appropriate. To prevent idle senders from being deleted, change the log destination setting. Existing senders can also be changed via API.

If we can remove a large number of existing senders for you, please ask.

Live now

We’ve tested this with a few customers and it’s made their Papertrail Dashboard a more accurate view of their environment with no effort. In this case, a simple decision does means less work.

As usual, please send any comments or suggestions. Enjoy!

Welcoming Larry Marburger to Papertrail

Posted by @troyd on

After more than 2 years of interacting with Larry Marburger as a Papertrail customer, technologist, entrepreneur, and trusted friend, I’m incredibly pleased to welcome him to Papertrail.

Many users know and use a service which Larry helped create, CloudApp (cl.ly), for capturing screenshots and sharing files. In fact, we used CloudApp at Papertrail long before I knew Larry personally.

Opportunities often arrive in unusual forms and our job is simply to try not to get in the way. Such was the case here. When CloudApp became a Papertrail customer, I said hello, and in the 2 years since, Larry has been a sounding board for Papertrail’s product ideas. Helping make these ideas a reality was the logical next step.

His in-the-trenches experience with API design, Ruby, metrics, and performance optimizations, coupled with his experience as an entrepreneur and a customer, let Larry see a unique end-to-end view: what’s truly useful, how it should work, and how it should be implemented.

Larry resides in Harrisburg, PA with his wife and two kids. In his spare time… wait, he has two kids. Larry does find time to help maintain Ruby Bundler, listen to podcasts, and partake in the official Papertrail morning beverage. You may see him at Folklore Coffee in Elizabethtown, PA.

We’ve been fortunate to work with Larry as a customer and advisor and we’re even more fortunate to welcome him to Papertrail as a contributor.

Free, un-copyrighted NDA and employment contracts

Posted by @troyd on

Papertrail’s staff grew in 2013, and as part of hiring employees, the company obtained employment agreements. These are fairly standard contracts that all technology companies use. If you’ve worked in the technology field, you’ve probably signed one.

We wanted to go a bit further, though. We got tired of not having easy-to-understand contracts and we want to start collaborating on something better. All stakeholders should be able to easily review, see changes to, discuss, and hopefully, edit and improve them.

Our stakeholders use git and Markdown. That’s why Papertrail’s employment agreement and NDA contract templates are now available on GitHub. They’re free and completely unencumbered under the Creative Commons Zero license.

Important: as the README states, Papertrail does not warrant that these are suitable for any purpose. These do not replace an attorney. Any use of these contracts is at your own risk and should be under the guidance of your own legal advisor.

Our goals:

  • make each party’s expectations obvious
  • improve the contracts over time. Not only are these contracts not set in stone, we fully expect them to change. Idea? Submit a pull request or issue.
  • provide other companies with a head start on, and chance to see the topics and changes which come up in, these agreements.

Fork, edit, and enjoy!

Self-service protocol options

Posted by @lmarburger on

If you’ve ever needed to log from systems or services which don’t support TLS encryption, check out the new log destination options available today.

Every Papertrail log destination has always supported sending logs over TCP with TLS encryption or UDP without encryption. Some systems and services can’t use TLS encryption and can only send logs in plain text TCP. This required manual intervention on our part to flip the switch to accept these logs. More people have been requesting this change so we added it as a self-service option giving you control over how we listen for logs from your systems.

Adding a new log destination or editing an existing one will present new options to enable or disable any of the supported syslog protocols: TCP with TLS encryption, plain text TCP, and plain text UDP. All options use the same port number regardless of the protocol and encryption used.

Examples

Here are a few scenarios where these new options will be of use to you.

  • Security: Enable only the TCP/TLS option and we won’t listen for logs sent over plain text.1

  • Fastly: Logs drained from Fastly can only be sent over plain text TCP. After creating a new destination to use with Fastly, edit it, check the TCP: Plain text option, and configure your Fastly account to log to this destination.

  • Multiple Senders: Several systems using the same log destination can be configured to log using the protocol and encryption that makes sense for that system. Use a single destination for all of your systems.

  1. Because UDP is connectionless, it’s possible for a system to send unencrypted logs over the network even if Papertrail isn’t configured to accept logs over UDP. Disabling UDP on a destination will drop those those logs and they won’t make it into Papertrail so you’ll know immediately that something isn’t configured correctly. 

Welcoming Leon Sodhi to Papertrail

Posted by @troyd on

We’re thrilled to officially welcome Leon Sodhi to Papertrail. Leon started in May, 2013 as Papertrail’s support engineer, so you may already have met him. Leon’s thoughtful disposition, attention to detail, and readiness to dive in and reproduce a problem have saved time for hundreds of customers.

Whether he’s helping identify an rsyslog configuration problem, patching NLog to properly handle multi-line events, discussing the pros and cons of encrypting logs with TLS, or using tcpdump to make quick work of unusual traffic, Leon is improving someone’s day.

Leon signs in from Ashford, Kent, England.

In his spare time, he recently built an electronic die. Hold a button down and a “die” (single dice) is rolled. The LEDs on the right show the result, like this.

He’s well on his way to running Linux on an 8-bit microcontroller, not unlike the original Nintendo.

We couldn’t be more thrilled or more fortunate to have Leon helping Papertrail users every day. If you come by Papertrail’s chat room, please introduce yourself and say hello.

Choose which email(s) you and others receive

Posted by @troyd on

As of today, Papertrail offers new ways to control which emails should go to which team members, such as to only send account usage updates to an operations team lead and credit card receipts to an accounting role account.

Team-wide controls are within Account. Your own preferences are duplicated on your Profile.

Team-wide email options

Here’s how it works.

Notifications

Papertrail sends two types of emails:

  • Usage. Examples: 90% of log data transfer; pay-as-you-go additional usage updates.

  • Billing. Examples: payment receipt; charge failure.

The specific emails and recipient controls depend on how you consume Papertrail’s service. For example, if Papertrail is not charging your credit card, billing email controls aren’t relevant and thus aren’t available.

Previously, usage notifications were sent to all users with full access to your Papertrail account. Billing notifications were sent to the first accountholder.

Common situations

  • Only send usage notifications to a few team members, not everyone.

    Within Account, visit Members. Enable or disable recipients.

  • Also send billing notifications to an accounting@ alias (which does not have a Papertrail account).

    Within Account, look for the “Purchases” tab. Add the accounting@example.com address. Optionally, set preferences for Papertrail accountholders in Members.

  • I want to change which emails I personally receive.

    Head over to your Papertrail Profile. This is an easy way to edit only your own options.

Personal email options

Questions? Ideas?

Let us know, and enjoy!

Smarter sorting in the Dashboard finder

Posted by @troyd on

Papertrail’s Dashboard finder now shows more specific matches first. Type any saved search, system, or group name. The most specific matches will reliably be the first results.

What changed?

image

Land on the Dashboard, press s (or click in the text box near the top of the page), and type the name of a group, saved search, or system. Matching saved searches, groups, and systems are shown. For searches, the finder also considers the name of a group that a saved search is associated with.

Until today, the results were sorted only by type, with saved searches first. However, saved searches were shown when the name of the related group matched, which led to a lot of matches.

As of today, Papertrail differentiates between matching resources (for example, the name of a saved search) and one-degree related resources (for example, the group that the matching search is related to).

Example

Imagine entering Mega Service in the finder. A group named “Mega Service” should always be shown first, well ahead of its searches (like “Important errors”) and slightly ahead of similarly-named groups (like “Staging Mega Service”). Now it is:

image

The current behavior is also retained, which means the finder still serves as a convenient account-wide search. For example, “Mega Service” will still match all 3 resources above. They’ll be shown after any more-specific matches, so now it’s also a fast path to an exact view.

We hope this makes the Dashboard finder a faster jumping-off point for more debugging sessions. Enjoy.