logs.papertrailapp.com TLS/SSL cert will change January 27

Posted by By telliott on January 6, 2016

Summary

The TLS/SSL certificate used by 1 of Papertrail’s syslog destinations,
logs.papertrailapp.com, will change on Wednesday, January 27, 2016. Some log senders which were configured before June, 2014 and are using TLS/SSL need to be modified to trust the new certificate.

We’ve tried to make this change as fast and painless as we know how to. It should take less than 5 minutes. If Papertrail’s team can save you time, please email support@papertrailapp.com.

Updates

2015-01-27: logs now presents the new wildcard certificate and full chain.
2015-01-25: Reminder email sent.
2016-01-20: 2-hour test performed. All customers with any senders which failed TLS handshaking were emailed.
Week of 2016-01-07: All customers who may be affected were emailed.

Does this affect me?

If this change may affect your systems, Papertrail will email you the week of January 4 and again before January 27.

This change only affects senders which meet all 4 of:

log to logs.papertrailapp.com (rather than logs2.papertrailapp.com or logs3.papertrailapp.com)
use TCP with TLS/SSL encryption (rather than UDP or cleartext TCP)
transmit logs with remote_syslog, rsyslog, or syslog-ng (rather than other daemons)
began logging to Papertrail prior to June, 2014

Only senders matching all of these constraints are affected. If 1 or more are not true, your senders are not affected.

Papertrail will email you the week of January 4 if any of your systems currently log to, or recently logged to, logs.papertrailapp.com.

What do I need to change?

remote_syslog

Download and install remote_syslog v0.16, released 2016-01-05:

.tar.gz (recommended), or
.rpm/.deb (new, in beta)

remote_syslog is a single self-contained program. To upgrade:

.tar.gz: uncompress the archive. Copy the new remote_syslog binary on top of the existing one, overwriting it.
.rpm/.deb: install the package.

Finally, restart remote_syslog. That’s it.

More: remote_syslog setup

rsyslog

First, see whether rsyslog was configured before June, 2014. Run: grep -r ActionSendStreamDriverPermittedPeer /etc/rsyslog*/*

grep may find a configuration directive like this: $ActionSendStreamDriverPermittedPeer *.papertrailapp.com. If this directive is already present, then the system was configured after June, 2014 and no change is needed.

If grep finds no matches, then the system needs 2 rsyslog configuration changes. Edit the file containing the Papertrail destination, usually /etc/rsyslog.conf, then:

1. Update trusted certificates

Find the line beginning with $DefaultNetstreamDriverCAFile. Download and save papertrail-bundle.pem in the location listed. For example, if the line reads:

    $DefaultNetstreamDriverCAFile /etc/syslog.papertrail.crt

.. save the new file to /etc/syslog.papertrail.crt location by running:

    sudo curl -o /etc/syslog.papertrail.crt https://papertrailapp.com/tools/papertrail-bundle.pem

Alternatively, save the new certificate file to a different location, then change the $DefaultNetstreamDriverCAFile line to point to that location.

2. Accept wildcard certificates

On the line below $DefaultNetstreamDriverCAFile, copy and paste this 1 line:

    $ActionSendStreamDriverPermittedPeer *.papertrailapp.com

3. Restart

Restart rsyslog with:

sudo killall -HUP rsyslog rsyslogd

More: rsyslog TLS setup.

syslog-ng

Follow steps 1 and 3 of syslog-ng setup. This will update the TLS certificates trusted by syslog-ng. Step 2 (configuration) has not changed and can be skipped.

When will this happen?

Please make the change above any time between now and January 26, preferably before January 20.

Papertrail will change certificates on Wednesday, January 27, 2016 during the US business day. With the configuration changes above, senders will experience no impact.

1 week prior (Wednesday, January 20): Papertrail will change to the new certificate for 2 hours, identify senders which do not reconnect, and then revert to the current certificate. I’ll send a second notification on Thursday, January 21 to customers with 1 or more senders which did not reconnect.
2 days prior (Monday, January 25): I’ll send one last email.

If this doesn’t affect you, I’m sorry for the inconvenience. Know that I’ve used all the information available to us to decide whether this may be relevant, and that we’re only taking this much care in order to prevent a service problem.

Can I test my systems?

Absolutely. First, we’re happy to review your configuration. Just reply and attach it.

Second, here’s how to verify that a sending system is updated:

Visit Destinations and click “Create log destination.”The new destination will be on a hostname which already presents the new certificate. It will already function the way that logs.papertrailapp.com will after January 27.
On an existing TLS-enabled system, change the rsyslog, remote_syslog, or syslog-ng to use the new log destination.
Visit Papertrail’s Dashboard and click the “All Systems” group.Scroll to this system’s name. You should now see 2 Papertrail entries for this system, not 1, since Papertrail will treat the logs sent to this new destination as a second system. If you do see 2 Papertrail entries, and the new entry has only a few minutes worth of logs, the system is configured correctly to accept the new certificate.
Change the system back, then visit Destinations to remove the test log destination.

Why is this necessary?

Papertrail’s first destination, logs.papertrailapp.com, presents its TLS certificates in an improper order, and prior to June, 2014, Papertrail’s setup instructions relied on that order. In June, 2014, the instructions were updated to work with both correctly-behaving newer destinations and the existing logs presentation. Papertrail’s original logs.papertrailapp.com TLS certificate will expire on February 6, 2016, so senders which were configured with the older instructions need to be able to accept a new certificate before then.

In addition, newer log destinations present a wildcard certificate (*.papertrailapp.com). rsyslog requires an explicit configuration directive to trust it. Since Papertrail did not use a wildcard certificate until June, 2014, rsyslog instances configured before then do not include this directive.

The changes above update senders configured prior to June, 2014 so they:

trust the same set of root certificate authorities as Mozilla does, so that future new certificates signed by trusted roots will work automatically.
trust Papertrail’s wildcard certificate. That way, logs can present the same wildcard certificate already presented by other destinations.

This was not caused by a security problem. The current configuration encrypts log data and verifies certificate trust as intended.

We’ve put a ton of effort into making encryption easy, but it’s not completely effortless. This is one case.

Questions

All of us at Papertrail want to make this as simple for you as we know how to. While there’s no way to avoid this change, if we can do anything to make it easier, we want to hear it. Please email support@papertrailapp.com.