We use Celery to manage/perform long running background operations (some take 30m+). To gracefully support continuous delivery, we want all workers to finish executing their existing tasks before fully shutting down. Celery handles this properly on a SIGTERM (stops pulling new messages from the queue and requeues any un-started jobs, but also waits until all active jobs complete).
However, Aptible appears to only provide a 10 second grace period between the SIGTERM and a forcible SIGKILL (“docker stop” default, as noted in this issue: Send TSTP Signal or set -t timeout parameter when killing process). This forced KILL results in any incomplete jobs being lost.
Is there any way to increase/alter the grace period used during container shutdown?