Debugging database down time

Hi all,

We’ve set up a pretty basic health check for our Postgres database that we have Pingdom checking every minute. (The endpoint just executes a single SELECT statement against the database and returns a 200 if it succeeds or a 500 if it doesn’t.) This particular health check has failed for 6 or so minutes each day for the past couple of days, and nothing interesting at all shows up in the Postgres log drain. What’s the best way to go about getting more useful info in that log drain aside from slow queries and the (very regular) ‘Incomplete startup packet’ messages that I’ve been told aren’t worth worrying about? It’d be lovely to even get access to the connection log. At this point, it’s difficult to tell if there’s something going wrong on the database server itself or if there’s a connectivity issue between it and the app server.

Thanks!
Logan

Hi there,

I’d suggest you start investigating this issue by looking at your application logs when you hit this error (presumably, your Pingdom check your app which in turn hits your database ;)). The issue might not be caused by the database at all, but even if it is, you’ll get a much clearer picture of what the issue is by looking at your application logs when you hit an error.

That being said, to answer your question, you can configure your Postgres logs however you want by connecting to your database (e.g. through aptible db:tunnel) and using ALTER SYSTEM SET followed by pg_reload_conf(). There’s more context here: https://www.aptible.com/blog/database-logs/, but note that Postgres is normally reasonably quiet about its activity by default.

Generally speaking, if you need to change the logging level, you can use a command line this (see the Postgres documentation. Note that we do configure Postgres to log at the INFO level by default, so you probably don’t need to change anything here:

ALTER SYSTEM SET log_min_messages = 'INFO';
SELECT pg_reload_conf();

In your case though, the logging level isn’t what controls the logging of new connections. Instead, this is controlled by the log_connections setting in Postgres (you’ll find that documented in the same place), so you’d want:

ALTER SYSTEM SET log_connections = true;
SELECT pg_reload_conf();

Feel free to open a support ticket at http://contact.aptible.com if you want us to take a look on our end.

Cheers,

1 Like