Handling github job cancellation signals

Demos for github job cancellation signal handling to show exactly how
it behaves. (Spoiler: badly).

Why is this needed? Because the github actions docs for this appear to be
nonexistent at time of writing, and the answer on the relevant github
"community" forum post
are nowhere near clear or detailed enough. (It's always possible I'm just bad
at searching, but you'd think the post above would link to the docs if that
were the case).

I was having issues with Terraform runs leaving lock files lying around on S3
state storage buckets when github jobs were cancelled. This shouldn't be
happening, because terraform will try to clean up and remove its lock file when
gracefully killed. Given the lack of usable github docs, I had to write a demo
to find out what exactly happens when a github actions job is cancelled.

TL;DR: unless special care is taken, child processes don't get any signal
delivered to them before they're hard-destroyed when a github actions job is
cancelled. As a workaround, if: always() blocks can be used to do necessary
cleanup steps as they run on cancel. Or for simple cases you can exec your
process, so it becomes the top-level process for a step and does receive
signals on cancel.

The demo code

A set of github actions demonstrate how github job cancellation signal handling
works. To try them you must run them via github actions, then cancel them using
the github actions UI.

• cancel-test-exec-child-ignore-sigquit.yaml:
  exec's a script that ignores SIGINT, SIGQUIT and
  SIGTERM. It shows that github delivers a SIGINT, waits 7.5s, delivers a
  SIGTERM, waits 2.5s, then presumably sends a SIGKILL. It then runs any
  if: always() steps after destroying all processes running in the killed
  step.

• cancel-test-exec-child.yaml:
  exec's the same script, but only ignores SIGINT, so the child process
  will terminate on the subsequent SIGTERM. This is more realistic, and the
  subsequent test cases do the same thing.

  Since it checks the process tree in the if: always() cleanup step, this
  test also shows that github destroys all processes under the step recursively
  before it begins any cleanup steps. It must be keeping track of all
  processes.

• cancel-test-shell-plain.yaml:
  Represents the "normal" case of a github actions step using a bash shell that
  runs a child process as a blocking command within the shell. You will see that
  the child process (the same script as the above demo) does not receive any
  SIGINT or SIGTERM. The bash leader process does, but you can't see that because
  bash defers trap execution until child process exit when blocking waiting for
  a child process, and the whole lot gets SIGKILL'd before the child process exits
  to return control to bash.

  This means that the workload running in the inner script got no chance to clean up
  its work.

• cancel-test-shell-sigfwd.yaml:
  Demonstrates that it is possible to use a top-level shell with job control
  enabled as a process-group leader that forwards signals to its child
  processes. It's ugly though. Because of deferred traps, every subcommand
  that needs a chance to handle signals must be run as a background job with
  & then waited for, and there's plenty of fiddling about to make it work.

  See comments in
  signal_forwarding_wrapper.sh for details.

• cancel-test-long-running-if-always.yaml:
  Explores what happens when an if: always() step takes too long or
  refuses to exit.

  It seems like github will let the cleanup run for about 4 minutes then kill
  it, initially with a SIGINT.

  Repeated cancels sent during the if: always() run appear to have no effect.

  Interestingly, it the job won't retain logs if the cleanup job doesn't exit
  within the overall job timeout, you can only see the logs if you were
  streaming them during the run.

• test-daemonize.yaml and test-daemonize-cancel.yaml:
  Shows that by detaching from the current session a process can remain running
  after the end of the containing step, or the cancellation of the step.

  So it's not possible to rely on Github to terminate everything running
  under a step; there could still be workloads running when your if: always()
  steps run after cancellation of work. You may need to ensure you record
  their pids and kill them yourself before handling cleanup.

  I presume they're destroyed when the containing job is cleaned up - I
  didn't bother giving the task telemetry to some public endpoint so it could
  report how long it stays running for.

Why child-process tasks don't get a chance to clean up on job cancel

Consider a simple job step like:

   - name: whatever
     shell: bash
     run: |
       my-long-running-task

You might expect that if the actions run is cancelled, my-long-running-task
would get some kind of signal to give it a chance to clean up before the whole
actions runner is destroyed. As if you'd pressed Ctrl-C in the shell, then
waited a bit, then pressed Ctrl-\ (break).

In reality, it exits (presumably on SIGKILL) without any chance to clean up.

On cancel, github actions delivers a SIGINT only to the top-level process
for the current step of each active job. Then 7.5s later it delivers a
SIGTERM, again to the top-level process only. 2.5s later it sends a SIGKILL
(presumably to everything in the process tree).

You'd think that's fine. But signals don't propagate down process trees,
so child processes running under the top-level step process won't see a signal
unless the top-level process explicit forwards it.

A typical Github actions job will be a run step with shell: bash that
invokes some task as a child process of the step's top-level shell. If you
cancel a job with this, github actions will signal the top pid (the shell) with
SIGINT.

Bash will behave as documented:

When Bash is running without job control enabled and receives SIGINT while
waiting for a foreground command, it waits until that foreground command
terminates and then decides what to do about the SIGINT [...]

The process will never get that SIGINT, so it'll never exit and bash never
gets to do anything. And you can't use a trap on SIGINT to forward signals
to the child process(es) either, because:

If Bash is waiting for a command to complete and receives a signal for which
a trap has been set, the trap will not be executed until the command
completes.

This issue isn't specific to bash, it's just a useful demo because it's the
widely used default for github actions.

What github actions should be doing

Instead of signalling the parent process, github actions should at least offer
the option of signalling all processes running under the current step.

A simple albeit imperfect way to do this is to spawn the bash shell as a
process group leader by setting the -m flag. Github actions would then
signal the whole process group for each active step when a job cancel request
is sent by sending a kill signal to the negated pid of the leader shell. This
works well for simple cases, but process groups aren't nestable, so if some
workload under the step creates its own process group it'll be orphaned and
won't receive the process-group signal unless its parent notices and propagates
it explicitly.

It'd be better still if it used linux sessions, pid namespaces, or other
suitable constructs that can properly contain all processes in the step and
signal them all at once.

While bash's usual awful-ness contributes to making this hard to get right,
GitHub's lack of docs or any means to configure the job signal handling to
deliver to a process group certainly makes it much worse.

Workarounds

Until/unless github improves their actions runners with configurable cancel
timeouts and the option to signal the whole process tree, there are a few
possible workarounds:

• exec any long-running tasks that might need a chance to respond to SIGINT
  or SIGTERM with cleanup steps before termination, so they become the top-level
  process in a step. Split series of such tasks into separate github actions
  steps instead of a single step with multiple sub-commands called by a shell.

  This'll work ok if your job can respond to SIGINT and clean up within 7.5s,
  or to SIGTERM and clean up within 2.5s. But if it's relying on network
  resources that's going to be iffy - and it probably is, since the runner
  itself is usually stateless.

• Have your command or the controlling shell write a state file to the runner
  working directory and remove it on clean exit. Then use an if: always() action
  to check for the state file and perform the cleanup actions if found.

  There is going to be a race between the task completing its work and deleting the
  state file, so you'll need to ensure that the cleanup action is idempotent.
  In other words, it must be safe to run the cleanup steps twice.

  For example, if you're unlocking something you should write a unique
  lock-instance-id to your state file. Then when you unlock it, you can ensure
  the unlock request gets silently ignored if the lock is currently locked with
  a different lock-instance-id. Handily, this is how Terraform force-unlock
  works.

  It's unclear what the rules for if: always() actions are during github
  actions cancellation. From a quick experiment it looks like you've got about
  4 minutes to complete cleanup, and if: always() steps ignore cancel
  requests.

• Hope that steps inside Docker containers work better?

  I haven't tested this yet but maybe github has saner cancel behaviour when
  a job runs in a user-defined container?

To investigate further

• How github behaves with jobs using containers
• Why github actions, which is a job control system, doesn't document
  fundamental and basic properties of job cancellation behaviour.

See also

• bash signal handling
• How to propagate a signal to child processes from a Bash script