Sane logging with papertrail


#1

Would love a pointer on how to set it up to logically group systems by app/process vs. by hostname, which is the default I’m getting (lots of container UUIDs that change on every deploy).


#2

@hylas I make a new drain per aptible environment/stack, that way I can tell the incoming to go into difference groups based on environment

Then each is tagged with {environment}-{service}, so you can use filters (and save them in papertrail) to get at the things you’re looking for

I’d recommend not thinking about hostname, it shouldn’t be applicable in a horizontal cloud system like this.


#3

That sounds like a decent approach. But in your main view, you still deal with the cruft of all the hostnames appearing in your list, or Papertrail then takes care of this? Just adding the drain without any add’l config (got distracted) gave me the following mess:


#4

You can make system groups, and then you can configure the drain to automatically put new systems into that group

That will let you do production vs dogfood vs. demo, etc.

Beyond that, I don’t spend much time on that system page. I head into the events and do something like

phs-production-web


#5

Speaking of papertrail, we’re finding that activity ssh’ed in on the rails console doesn’t get sent. :frowning:

Anybody have any easy solutions for this? best I can tell is it is sending STDOUT to the screen, and not to the log drain; any chance anybody has run into that and has a solution?


#6

@Philonous right now we don’t send logs from your aptible ssh sessions to your log drains. Part of the rationale is that it’s very common for PHI to appear in REPL output, and we don’t have a way to opt out of sending logs on a per-session basis, or to disable it for SSH entirely, in the case where a non-BAA-signing log drain (like Papertrail) is being used.

This is on our roadmap though: https://github.com/aptible/support/issues/155


#7

So in our papertrail logs, each event is associated with an container ID. Is there a way to access this container id from within the container/app?


Container unique id for metrics reporting
#8

@hylas yeah, it’s just hostname (as in: calling the hostname command)

Also, we supply a couple ENV values

  • APTIBLE_PROCESS_TYPE: e.g. web, worker, etc.
  • APTIBLE_PROCESS_INDEX: (zero-based) index of the container within the process type

E.g. web0 would have APTIBLE_PROCESS_TYPE=web APTIBLE_PROCESS_INDEX=0


#9

Good news! Logging from SSH sessions is now available: https://www.aptible.com/changelog/capture-ssh-session-logs-with-log-drains/