Hi, we have a cron job which runs some heavy processing. Lately it’s been under heavier load, so we noticed some jobs backed up. After tailing the logs, we found out its because Aptible terminates the service when it reaches memory the limit, and then restarts the service.
Is there a way to notified of this termination?
example:
2017-05-15T20:31:21.826Z [cron asdfasdfsdf - aptible]: container exceeded its memory allocation
2017-05-15T20:31:21.827Z [crona asfasfasf - aptible]: container is restarting:
We’ve been using Sumologic as a log drain (they’re inexpensive as far as these services go, and they will sign a BAA without committing to an “enterprise” pricing plan). You can setup real-time alerts using scheduled searches on Sumologic.
1 Like
As Alan noted: alerting on your logs is the way to go here: the container exceeded its memory allocation message is sent to all Log Drains (on non-legacy infrastructure, aptible logs is simply a Log Drain: anything you see there is visible in other Log Drains and vice-versa).
Note that if you don’t want to set up a provider like Sumo Logic to handle these, you could opt to drain those logs to an app hosted on Aptible (via a HTTPS Log Drain) and process them there.
Cheers,
1 Like
This also sounds like a potential fit for a queue, unless the processes in the crontab must run in a specific order or operate on local (ephemeral) objects. If order is not important, you could at some point refactor to post the commands to a queue which you can implement with the Aptible Redis database container, which is simple to set up, fast and does a nice job. Supervisor can spawn N workers to pull the queued jobs on a separate container which would also leverage multi-core or you could pick a job manager that is distributed so you could run different cron tasks on separate containers as you scale.