xargs -P and find -print0: Running Shell Jobs in Parallel

Last month the nightly archive job on our log box crossed 41,000 files and became something I had to babysit. Every rotated log under /var/log/archive gets compressed at level 6 before it moves to the cold tier, and the loop I wrote two years ago does it one file at a time. On a box with twelve idle cores.

Here is that loop. The fix is short; the traps around it cost me an afternoon.

#!/usr/bin/env bash
find /var/log/archive -name '*.log' -print0 |
while IFS= read -r -d '' f; do
  gzip -6 "$f" 2>/dev/null
done

It works, and it is 41,000 process launches in a row: about six hours.

Why You Can't Just Pipe Into a Command

My first attempt at it was worse:

find /var/log/archive -name '*.log' | rm

That exits 0 and deletes nothing. rm never reads standard input, and neither do gzip, cp or chmod. They take paths as arguments. A pipe hands them a stream of bytes, which they ignore.

xargs is the adapter: it reads items off the pipe and builds command lines, because the argument list has a size limit. On this Mac, sysctl -n kern.argmax reports 1048576, and going over it looks like this:

$ /bin/echo $(printf 'x%.0s' $(seq 1 2000000))
bash: /bin/echo: Argument list too long

That is E2BIG, the error that sends people to the xargs manual at 2am.

The Whitespace Split

By default xargs splits input on any whitespace. Two files with spaces in their names:

$ find . -name '*.txt' | xargs -n 1 echo TOKEN:
TOKEN: ./my
TOKEN: file
TOKEN: 1.txt
TOKEN: ./my
TOKEN: file
TOKEN: 2.txt

Two files in, six tokens out. Had the command been mv or chmod, three of those tokens would be pointing at paths that do not exist. The fix is to make find emit NUL bytes instead of newlines, since filenames cannot contain a NUL byte:

$ find . -name '*.txt' -print0 | xargs -0 -n 1 echo TOKEN:
TOKEN: ./my file 1.txt
TOKEN: ./my file 2.txt

-print0 has been in GNU find for ages and POSIX added it in Issue 8 in 2024, so -print0 | xargs -0 is not the careful version of a command, it is the default one. It also stops xargs from treating quotes and backslashes as special.

Dry runs deserve a warning. This one wasted twenty minutes of my life:

$ find . -name '*.txt' -print0 | xargs -0 echo rm
rm ./my file 1.txt ./my file 2.txt

Drop the -print0 | xargs -0 and the line is identical, while the real command gets six broken paths. echo joins its arguments with spaces, so it flattens the exact problem you are trying to spot. -t is the honest flag: it prints each command line before running it.

Parallel, Which Is the Actual Point

-P is why I rewrote the job. The manual says to pair it with -n or -L, otherwise chances are only one exec happens, which is a sentence you only understand after watching a "parallel" run take exactly as long as the serial one. -P 0 means as many processes as possible, which I would not do on a shared box.

JOBS=$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)
ARCHIVE=/var/log/archive

find "$ARCHIVE" -name '*.log' -print0 |
  xargs -0 -r -n 1 -P "$JOBS" gzip -6

Six hours became forty minutes. Not the twelve times speedup the core count suggests, since all twelve workers push through the same disk, but I will take it. getconf _NPROCESSORS_ONLN prints 12 here and on the Linux runners. nproc is the Linux shortcut and sysctl -n hw.ncpu the macOS one; neither exists on both.

One more thing about -P. If two children print to stdout, the manual says the output arrives in an indeterminate order and very likely mixed up. The gzip warnings survived that. A script printing a status line per file did not, and the fix was one output file per child.

The -I Flag Quietly Undoes the Batching

-I {} reads nicely and costs more than it looks like, because it implies -L 1: one invocation per input line, batching gone. GNU xargs also treats -L, -I and -n as mutually exclusive, keeps whichever came last, and warns on stderr, easy to miss in CI output. The exception is -n1 after -I, ignored because it would not conflict.

When a command wants the item somewhere other than the end of the line, look for a flag that takes a destination before reaching for -I. mv has -t, and this keeps the batching:

find . -name '*.bak' -print0 | xargs -0 mv -t /tmp/backups

Exit Codes, and the Failure You Don't See

GNU documents 123 when an invocation exits with anything other than 0 or 255, 124 for 255, 125 for a signal kill, 126 for cannot run, 127 for not found. My Mac's BSD xargs returned 1 when I ran a child that exited 1, not 123, so the portable reading of that number is "non-zero" and checking for 123 is a Linux habit.

The empty input case is the one that bit me. On Linux, xargs runs the command once even when there is no input at all, unless you pass -r. macOS skips it. Same script, two behaviors, and the Linux one is dangerous, because your command runs with no arguments whatsoever. My wrapper read input="$1", and when the runner called it with an empty find result the variable was blank and it globbed the working directory instead. Nothing was lost that day. The log line said success, which is what annoyed me.

-r is accepted without complaint by the macOS xargs I tested and is not optional on Linux, so leave it in. Pair it with set -euo pipefail, otherwise the pipeline's exit status is just the last command's and the xargs number never reaches you. I wrote about the rest of strict mode and its sharp edges earlier.

What I Keep Now

For per-item work where items are independent, xargs wins. It runs several at a time, and with -print0 and -r it fails loudly instead of guessing. When the loop body needs branching, shared counters, or variables that survive between iterations, a while read loop is still the right tool and I will not pretend otherwise.

The version above lives in Snippet Ark, JOBS line and -r already in it, so the next batch job starts from the script that works, not the one that hides its own errors.