When troubleshooting, a single search can become a theme with variations. Even if it’s not worth saving, it’s worth remembering, for a while. Until today, Papertrail didn’t make getting back to recent searches especially easy, but that’s changing.

Today we’re excited to release a new feature that provides user-specific access to search history.

How it works

There are two ways to access search history in Papertrail’s Event Viewer:

By clicking

The magnifying glass in the search input now toggles the search history view:

Once the search history is opened, click on the search you want to perform.

Keyboard shortcuts

We designed search history to be an extension of the search input, so it felt natural to support a keyboard workflow.

When the search box has focus, press ↑ to open the search history. Once opened, the list can be navigated using the ↑ or ↓ arrow keys. When the right search is highlighted, press Enter to execute it.

Modifying a recent search

When a query is close, but requires a small tweak, it can be tedious to rewrite the entire query just for that little adjustment. With search history, any recent search can be copied down to the search box for modification.

Saving a recent search

Right now, a search can only be saved in the event viewer if it’s currently active. With search history, a recent search can be saved any time, without needing to load it first. This keeps the focus on troubleshooting, without interruptions to save searches.

What do you think?

Not all searches need to be saved, but that doesn’t mean those searches aren’t important. With search history, they’re only an ↑ arrow away.

We hope you find search history useful. Give it a try, and let us know if you have any questions or feedback. Enjoy!

We’ve been working on a few new features that will dramatically increase the speed at which logs are searched and enhance visibility into log volume. These features are immediately available to new customers and will be rolled out to existing customers in the coming weeks.

Lightning Search changes the way Papertrail™ stores and searches logs, resulting in a dramatic increase in search speed. This new architecture also enables us to release a new feature: Log Velocity Analytics. Understanding log volume has never been easier. While viewing logs, click on the graph icon, and visually explore log data. The graph will reflect the current search, group or system.

Whether you’re trend spotting over a week, or diagnosing a spike in the last 10 minutes, Log Velocity Analytics reduces the time needed to understand the data in your logs.

Papertrail now officially supports automated setup of the remote_syslog2 logging daemon with Chef, Puppet, and Salt:

• chef-papertrail
• puppet-papertrail
• salt-papertrail

All three support Ubuntu, Amazon Linux, CentOS/RHEL, and Debian 9. (A previous version of this post mentioned an upstream issue with Puppet and Debian 9 that has now been resolved.)

The Chef cookbook supports 12.21.x and 13.x, and the Puppet module has been tested with 4.x and 5.x.

Get set up easily by providing:

• the account’s log destination information
• a list of files to monitor
• any desired config options for typical remote_syslog2 operation

If your config strategy of choice is cookbooks, modules, or formulas, try these out. Tell us if you run into any speed bumps.

Papertrail has had the ability to alert on searches that match events for years, but what about when they don’t? When a cron job, backup, or other recurring job doesn’t run, it’s not easy to notice the absence of an expected message. But now, Papertrail can do the noticing for you. Today we’re excited to release inactivity alerts, offering the ability to alert when searches don’t match events.

Set up an inactivity alert

From the create/edit alert form, choose “Trigger when no new events match”

Once saved, the alert will send notifications when there are no matching events within the chosen time period. Use this for:

• cron jobs
• background jobs which should run nearly all the time, like system monitors/pollers and database or offsite backups
• lower-frequency scheduled jobs, like nightly billing

Try it

If you have cron jobs, backup jobs, or other recurring or scheduled jobs, they almost certainly already generate logs. Here’s how to have Papertrail tell you when they don’t run or run but don’t complete successfully:

1. Search for the message emitted when a cron job finishes successfully (example: cron)
2. Click “Save Search”
3. Attach an alert, like to notify a Slack channel or send an email.

No logs? No problem.

Very rarely, a recurring job doesn’t generate log messages on its own. For those, use the shell && operator and logger to generate a post-success log message. For example, ./run-billing-job && logger "billing succeeded" will send the message billing succeeded to syslog if and only if run-billing-job finishes with a successful exit code. Use "billing succeeded" as the Papertrail search.

What do you think?

Give inactivity alerts a try and if you have questions or feedback, let us know.

Ever wanted to quickly see which systems haven’t sent logs recently? Now it’s as easy as checking a traffic light. Visit the Dashboard and click a group name, then scan the list of systems:

• Green-light systems are currently sending logs
• Yellow-light systems aren’t currently sending logs, but have sent logs in the last 24 hours
• Red-light systems haven’t sent logs in the last 24 hours

Text on the right shows when Papertrail last saw logs.

It’s an easy way to make sure critical systems are still logging after a deployment or upgrade.

Try it out, and if you see anything unusual, or just want to opine on the intervals or colors, tell us.

Today we’re excited to release two new keyboard shortcuts within the event viewer:

• highlight and link to multiple events
• retain the current search query when clicking

Highlight and link to multiple events

When a series of events is relevant, it can be useful to share those events with teammates. This is now possible.

Holding Shift will put the event viewer into “selection mode.” While holding down Shift:

• Start a selection by clicking the button next to an event.
• Select a range by clicking a selection button above or below an existing selected event.

The browser URL will update to indicate which events are selected. From there it’s as easy as copying and pasting the link. Clear a selection by pressing Esc.

Retain search query when clicking

Extending the idea of flexible context to the entire event viewer, holding Alt while clicking a link will retain (instead of replace) the current search query. This works for orange and blue context links as well as click-to-search.

To see these and all other keyboard shortcuts, press ? while in the event viewer. Try out the new shortcuts and as always, let us know if we can do better. Enjoy!

Papertrail’s search alerts are great, but what happens when you need a specialized integration, or want to grab something other than raw messages and counts – like particular fields from a message, or data to analyze later?

Now, you can invoke a Zapier action using a webhook trigger, which can then perform any desired action in Zapier. This example Zap setup sends data on printer service behavior to a Google Sheet for later analysis.

Set up the Zap

Sign up for Zapier, or log in.

Create a new Zap. Under Built-In Apps, select Webhooks by Zapier, then Catch Hook, and save the new Zap.

In the Pick off a Child Key dialog that appears:

enter payload.events to get to the event details and Continue to show the Zap’s webhook URL.

Set up the alert

Create a saved search to find the lines of interest, then add a Zapier alert integration. Grab the webhook URL and paste it into the alert, then save.

Process data

Create a Google spreadsheet with column names for the relevant fields: received_at, source_name, program, message.

and test the data-sending step by clicking Send test data on the Papertrail alert, and in Zapier, clicking OK, I did this.

Once the test has succeeded, set up the action. Select the Google Sheets app, then Create Spreadsheet Row. Select the spreadsheet and worksheet created earlier, and fill in the fields with selections from the payload.

Voila! A spreadsheet that will dynamically update with details from the matched events when the Papertrail alert fires.

If custom alerts going right into your app of choice sounds great, give it a try and let us know your thoughts.

Starting today, it’s possible to grant a user access to logs from certain senders/groups (within the same Papertrail organization). Additionally, we’ve added specific permissions for managing users, changing plans, and purging logs.

Here’s an example, where an administrator is changed to have ready-only access to certain groups:

What’s possible?

Papertrail’s granular access control and permissions allow:

• Companies to segregate access by responsible team, like granting access to logs from a staging environment or a specific product.
• Consultants and hosting providers to provide limited access to many customers, while still managing all logs themselves.
• Admins or accounting teams to handle less-common changes like adding users, changing plans, and purging logs.

These new permissions keep access clean within a single organization and may reduce the need for multiple organizations. Some may still benefit from having multiple organizations, or a combination of both multiple organizations and granular organization-specific permissions.

Give it a try

To change permissions, visit the Members section. And as always, we’d appreciate hearing your ideas.

Today we’re excited to release new ways to act on specific events, including seeing surrounding and related context, copying a deep link URL, and transitioning to the command-line.

With this new feature, event actions, one can:

• Link to an event in the same context you’re viewing. An easy way to share an event with a teammate or save it for reference.

• Transition to the Papertrail CLI at the same point the event occurred.

• Show an event within a different context, like a specific system, program, or group. Switching contexts allows quick examination of multi-line events or multi-system incidents.

Because text selection is often an integral part of log-based troubleshooting, the event actions “+” button doesn’t change text selection behavior. Select text as you normally do, even directly over the button, and it’ll work as normal.

Flexible context

In most situations, showing surrounding context means removing the current search query. It’s a bit like zooming out: show me a specific event in the context of all events from a given log sender, program, sender and program, or group. This makes it possible to transition between any set of related logs without losing the specific message you’re interested in.

When you’re certain that all relevant events match your current search, the context links can show surrounding context while retaining the current search query.

Imagine I’m looking at events matching the search format=html. I want to see a specific event in the context of a different set of events that also contain format=html. In this case, I’m interested in seeing matching events from a specific program (app/web.1):

The next time you need more context around an event, try out event actions and let us know if you have feedback. Enjoy!

Wouldn’t it be great if you could drill down into your logs just by clicking? Today, we’re excited to release a feature that lets you do exactly that. We call it click-to-search.

Here are a few examples of how this will make life easier:

• Web access logs: Each line in your access log contains an IP address. To click the address to see other lines containing the same address, enable “IP Addresses”.
• Custom app logs: Your custom application logs contain User IDs with the format “user_id=1234”. Creating a custom clickable element with the regular expression user_id=\d+ would make the string “user_id=1234” clickable.

Papertrail offers the following clickable elements out of the box:

• IP addresses (enabled by default)
• Email addresses
• GUID / UUID
• Period separated words (domains, file names, etc)

For these items, just click a checkbox to activate the elements that apply to your logs.

Since all logs are different, we’ve made it simple to create your own custom clickable elements:

Click-to-search will improve troubleshooting flow and make log filtering easier. We hope you find it useful, and we always welcome feedback on how to make it better. Enjoy!

Summary

Update: The new DNS records are now active.

The DNS records for Papertrail’s first four log destinations (logs.papertrailapp.com, logs2.papertrailapp.com, logs3.papertrailapp.com, and logs4.papertrailapp.com) will change on Tuesday, December 20, 2016. The new IP addresses will be in the CIDR block 169.46.82.160/27.

Important: Papertrail will continue accepting log messages sent to the old IPs.

Does this affect me?

Probably not. Loggers will continue logging to the old IPs until they’re restarted, at which time a new DNS lookup will take place. However, if the network uses IP-based egress filtering, the egress filters will need to include the new addresses by Tuesday. (Very few networks filter outbound traffic in this way.)

What if we aren’t using DNS?

For any systems sending logs directly to an IP address, no action is needed. The log destinations will continue listening on the old IPs until further notice.

Questions

Please email support@papertrailapp.com if there’s anything we’ve missed.

Papertrail’s event viewer displays the full text of your events including the timestamp, log sender, and program. Sometimes, one log element can get in the way of what you’re looking for.

We’ve added a new way to control how messages appear, as a link labeled “Options”:

Clicking the Options link exposes a menu of display preferences that can truncate messages for skim-ability and show or hide 3 different log elements.

Truncate Message

Enabling “Truncate Message” will trim log messages to a single line. When messages are truncated, clicking any message will show the full message again.

This option can be beneficial when:

1. many logs use similar or identical formatting, and the most important details are near the beginning of the message
2. viewing logs on a small screen or window

Hide Time, System, Program

These options will hide any combination of time, log sender, and program. Once hidden, the label is collapsed to a dot and hovering over that dot shows a tooltip of the hidden label.

In addition to maximizing space, these options can be handy when logs:

1. have long or low/no-value log senders or programs
2. have an implied date (or a timestamp already included in the log message)
3. are using a W3C, or similar, character-offset-based log format

Options Persistence

All display options will persist for the current browser window/tab. As a result, multiple instances of the event viewer can run in different tabs and the options can be tailored for each tab.

Questions

If you have any ideas or questions about this feature, please get in touch.

Summary: Navigate to any log view from anywhere by clicking the new icon in the upper left corner.

What changed?

The dashboard has always had a way to quickly find a specific saved search, group, or system:

But the reality is, you might not be on the dashboard when you want to find something. Well, as of today, you no longer have to be. There’s now a new search box at the top left of every page:

The same “Hit s and type” shortcut works as it always has on the dashboard, but is now accessible from anywhere within Papertrail.

Customers with only a few searches, groups, and senders don’t even need to type. Just click the icon or press the s shortcut to see all available log views:

Oh! And there’s a bonus: the new finder brought edit links along with it:

Saving unnecessary clicks and page loads is important, especially when troubleshooting a problem. Every second counts. We hope this change makes finding what you need—and when you need it—much more efficient.

As always, let us know if you have questions or ideas for a better finder.

We’re excited to release some subtle but powerful updates to Papertrail’s event viewer that make searching and sharing logs much easier:

• Searches now stay centered around the time you’re looking at, so step-by-step troubleshooting is faster.
• Event viewer URLs now link to exact positions, so colleagues always see exactly what you see.

Never lose your spot

When searching through logs, it’s common to start with a broad search and gradually edit the search query to refine the results. Previously, a refined search would start searching again from “now” even if you had scrolled to results from the past.

Now when you edit an existing search query, the results will be based on the time of the events you are currently viewing. This should give you a quicker and more accurate search experience. Of course when you do need to manually set a search time, that option is still available.

You see what I see

When all your logs are in one place, it’s easy to link to and share important events. We’ve made this as simple as copying and pasting the URL of your search. However, in searches that returned a lot of events, the URL might not indicate exactly what events you were looking at when sharing.

Now, you can share an event viewer URL with your team in confidence, knowing that whoever visits that URL will see the same set of events, in the same position, as was shown when the URL was generated.

How does it work?

These changes are automatically enabled in the event viewer. Keep doing what you’re doing! For example, when you perform a search, then scroll to a different position, and then perform a second search, you’ll be at the same time. And if you copy an event viewer URL and return to it, you’ll be looking at exactly the same log message.

What do you think?

These are subtle changes from how our event viewer has worked in the past, but we hope you’ll find they feel quite natural and improve your experience with Papertrail. Try it out and please let us know what you think. Thanks!

On Thursday, April 14, 2016, Papertrail will deploy a new SHA-2 (SHA-256) TLS/SSL certificate for its syslog destinations, replacing the current SHA-1 certificate.

Update 2016-04-25

On April 14, Papertrail deployed a new SHA-2 certificate, discovered that older versions of remote_syslog2 (0.13 and prior) did not accept the certificate, and reverted to the prior SHA-1 certificate.

Part of our job is to be our customers’ eyes and ears for changes like this. To that end, Papertrail will:

• present the new SHA-2 certificate for about 6 hours on April 27, 2016
• log all failed connection attempts due to TLS negotiation failures
• revert to the existing SHA-1 certificate
• email affected customers

The certificate will be deployed permanently on ​*Wednesday, May 4​*, 2016.

If we can answer any questions or help, let us know.

Update 2016-04-14

During the migration, we were reminded that older versions of remote_syslog2 pre v0.14 don’t support SHA-2. Based on this new information, we are reverting the TLS certificate and will post a revised deployment plan next week.

Impact

For nearly all senders and all common log senders (including remote_syslog2 and rsyslog), this will be a non-event. OpenSSL has supported SHA-2 since 0.9.7m and enabled SHA-2 by default beginning with 0.9.8l (released on 2009-11-05).

The only senders which Papertrail knows do not accept SHA-2 (SHA-256) certificates are those running:

• Windows 2003 SP2 without KB 968730 or 938397
• Windows XP SP2 or older (SP3 is not affected)

Windows 8, 7, Vista, Server 2012, and Server 2008 are not affected.

Why is this necessary?

The Wikipedia entry for SHA-2 explains the reason for this change:

Although (as of 2015) no example of a SHA-1 collision has been published yet, the security margin left by SHA-1 is weaker than intended, and its use is therefore no longer recommended for applications that depend on collision resistance, such as digital signatures.

Papertrail’s Web site already serves a SHA-2 certificate. This change only affects Papertrail’s syslog endpoints.

Questions

If we can help test an old device or otherwise save you time, we’re at your service: support@papertrailapp.com.