[{"data":1,"prerenderedAt":6},["ShallowReactive",2],{"post-content-bash-trap-exit-cleanup-exit-code":3},{"content":4,"lastModified":5},"\u003Cp>Our auto-publish script stages the build output in a temporary directory, uploads it to Cloudflare Pages, and cleans up on the last line. The last line is the part that never runs. When a deploy takes longer than the CI job allows, the runner kills the script and the directory stays behind, along with whatever \u003Ccode>wrangler\u003C\u002Fcode> was still writing into it.\u003C\u002Fp>\n\n\u003Cp>So I added the one-line fix bash gives you for this, directly under the \u003Ccode>mktemp\u003C\u002Fcode>:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">work=$(mktemp -d)\ntrap 'rm -rf \"$work\"' EXIT\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Disk usage on the runner went back to normal, which felt like the end of it. Then a deploy was killed, the trap did its job, and there was nothing in the logs to say the run had gone badly.\u003C\u002Fp>\n\n\u003Cimg src=\"https:\u002F\u002Fimages.unsplash.com\u002Fphoto-1518432031352-d6fc5c10da5a?auto=format&fit=crop&w=1200&q=80\" alt=\"A terminal window mid-download, with a transfer progress table above a process list showing running jobs and their process IDs\" loading=\"lazy\" \u002F>\n\n\u003Ch2>The trap runs on a SIGTERM, and $? is 0\u003C\u002Fh2>\n\n\u003Cp>Reduce it to the part that matters:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">#!\u002Fbin\u002Fbash\ncleanup() { echo \"trap saw $?\"; }\ntrap cleanup EXIT\nsleep 60\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Now send it SIGTERM, which is what a CI runner or systemd sends when it has decided you are done:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">$ .\u002Fsleepy.sh & pid=$!; sleep 1; kill -TERM $pid; wait $pid; echo \"process exited $?\"\ntrap saw 0\nprocess exited 143\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The handler fired without me ever trapping TERM, which is worth knowing on its own. The number it printed is the problem: the process left with 143, the status a shell reports for a process killed by signal 15, while the trap saw 0.\u003C\u002Fp>\n\n\u003Cp>The advice you find everywhere, and which I follow, is to capture \u003Ccode>$?\u003C\u002Fcode> on the first line of the handler so you keep the real status. It is correct, and I verified it: a script ending in \u003Ccode>exit 7\u003C\u002Fcode> shows 7 in the handler. It just does not hold when a signal did the killing. The 143 comes from the signal, not from anything the script decided.\u003C\u002Fp>\n\n\u003Cp>For me that meant the guard that emails me about failed deploys stayed silent on exactly the runs I care about. A killed deploy filed itself as a success.\u003C\u002Fp>\n\n\u003Ch2>And it fires before your child process dies\u003C\u002Fh2>\n\n\u003Cp>The signal goes to the script. The command it is running is a separate process that gets nothing, and bash is in a hurry to leave.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">$ pgrep -f 'sleep 60' | wc -l\n0\n$ .\u002Fsleepy.sh & pid=$!; sleep 1; kill -TERM $pid; wait $pid\ntrap saw 0\n$ pgrep -f 'sleep 60' | wc -l\n1\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The trap ran. The work was still going afterwards. bash executed the handler and exited immediately, orphaning the child. So a handler whose job is \u003Ccode>rm -rf \"$work\"\u003C\u002Fcode> removes a directory that a live process is still writing into. Three runs of that test left three orphaned sleeps, and nothing escalates them.\u003C\u002Fp>\n\n\u003Cp>That is the part I had backwards. I thought of the trap as a wall at the end of the script, with everything already finished on the far side.\u003C\u002Fp>\n\n\u003Ch2>Trap the signal and the timing moves\u003C\u002Fh2>\n\n\u003Cp>The obvious fix is to stop letting the signal do whatever it wants. Note it, then exit on purpose. That works, and it moves the problem instead of removing it.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">#!\u002Fbin\u002Fbash\ntrap 'echo \"handler at $(date +%S)\"' TERM\necho \"started at $(date +%S)\"\nsleep 6\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Start it, send TERM a fraction of a second later, and watch the handler:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">started at 42\nhandler at 48\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>bash does not run a trap handler while it is waiting on a foreground command. It notes the signal and runs the handler once that command returns. A graceful shutdown built this way waits for whatever is running at that moment, and the six seconds above could just as easily be a ten-minute upload.\u003C\u002Fp>\n\n\u003Cp>There is a second edge here. If the handler never calls \u003Ccode>exit\u003C\u002Fcode>, the script carries on to the end of the file and exits 0. Mine did, which turned SIGTERM into a no-op.\u003C\u002Fp>\n\n\u003Ch2>Background the work, then wait on it\u003C\u002Fh2>\n\n\u003Cp>The exit code, the timing and the orphan all have the same root: the script does not own the process doing the work. Background the long command, keep the pid, and wait on it. \u003Ccode>wait\u003C\u002Fcode> is interruptible, and that is the entire trick.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">#!\u002Fbin\u002Fbash\nset -euo pipefail\n\nwork=$(mktemp -d)\nchild=\"\"\nsig=\"\"\n\non_signal() { sig=\"$1\"; }\ntrap 'on_signal INT' INT\ntrap 'on_signal TERM' TERM\n\ncleanup() {\n  local rc=$?\n  trap - EXIT INT TERM\n  if [[ -n $child ]] && kill -0 \"$child\" 2>\u002Fdev\u002Fnull; then\n    kill -TERM \"$child\" 2>\u002Fdev\u002Fnull || true\n    wait \"$child\" 2>\u002Fdev\u002Fnull || true\n  fi\n  rm -rf \"$work\"\n  (( rc == 0 )) || echo \"exit $rc${sig:+ after $sig}\" >&2\n  exit \"$rc\"\n}\ntrap cleanup EXIT\n\nwrangler pages deploy \"$work\" & child=$!\nwait \"$child\"\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Against the same test, the handler now runs as soon as the signal arrives, the child is dead before the directory is removed, and the process exits 143, which is the number the trap sees. When a trapped signal arrives during \u003Ccode>wait\u003C\u002Fcode>, the builtin gives up immediately with a status above 128. \u003Ccode>set -e\u003C\u002Fcode> ends the script there, and the trap finally has a real number to compare against.\u003C\u002Fp>\n\n\u003Ch2>The same script was hiding a second success\u003C\u002Fh2>\n\n\u003Cp>With the exit codes finally visible, the other reason my script never reported a bad run was easy to spot. Every long step pipes through \u003Ccode>tee\u003C\u002Fcode> so the log file keeps a copy:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">set -e\nnode scripts\u002Fgenerate-blog-post.js 2>&1 | tee -a \"$LOG_FILE\"\n\nif [[ $? -ne 0 ]]; then\n  exit 1\nfi\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Two problems stacked on two lines. Without \u003Ccode>pipefail\u003C\u002Fcode>, that pipeline reports \u003Ccode>tee\u003C\u002Fcode>'s status, so \u003Ccode>set -e\u003C\u002Fcode> has nothing to react to. And by the time \u003Ccode>$?\u003C\u002Fcode> is read on the following line, it is still \u003Ccode>tee\u003C\u002Fcode>'s status. That check has never once been true.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">$ false | tee \u002Fdev\u002Fnull >\u002Fdev\u002Fnull; echo \"PIPESTATUS=${PIPESTATUS[*]} status=$?\"\nPIPESTATUS=1 0 status=0\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The 1 is right there in the array, one variable away from the code that would act on it. Add \u003Ccode>set -o pipefail\u003C\u002Fcode> at the top and the pipeline reports the generator's status, \u003Ccode>set -e\u003C\u002Fcode> ends the script, and the trap's exit code starts meaning something.\u003C\u002Fp>\n\n\u003Cp>What I keep coming back to is that a trap is not \u003Ccode>try\u002Ffinally\u003C\u002Fcode>. It is a hook the shell calls while it is already on the way out, at a moment when it may have lost track of its children. Treating it as cleanup that always tells the truth is how a killed deploy gets filed as a success.\u003C\u002Fp>\n\n\u003Cp>The skeleton lives in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> with the measured outputs pasted into the note beside it, because I will not remember which line matters in six months. It sits next to the \u003Ca href=\"\u002Fposts\u002Fbash-strict-mode-set-euo-pipefail\u002F\">strict mode notes\u003C\u002Fa> and the piece on \u003Ca href=\"\u002Fposts\u002Fjq-exit-codes-bash-scripts\u002F\">jq exit codes\u003C\u002Fa>: bash does tell you what happened, and reading the wrong value is how you end up believing it.\u003C\u002Fp>\n","2026-10-10",1791775771993]