All the bugs that are fit to print
My home server’s deploy failed at step one, and every step after it reported success. Here’s how a failure goes missing in a deploy chain, and a copy-paste script that refuses to call nothing a success.
I deployed a small website to my home server. It pulled, built and rolled out, Kubernetes reported success, and nothing changed. git pull had refused to overwrite three hand-made changes on the server, and its failure never reached the && that should have stopped everything; the likeliest culprit is a pipe through tail. The fix: a deploy script that pulls on its own line under set -euo pipefail, then makes four checks: a clean checkout, the commit, a new image and the pods. Plus two smaller silent failures: a zsh rm -f, and a backup behind a misleading rclone error.
git pull on a server can fail while every step after it succeeds, and how to spot it.set -euo pipefail changes.$del, and a misleading backup error.Anyone who deploys to a home server, a Raspberry Pi or a single VPS by SSH-ing in, pulling and building, and anyone who writes deploy scripts.
Level: intermediate. You’ll want to be comfortable with git and a shell. Kubernetes basics help, and there’s a Docker Compose variant.
One afternoon in August I committed a clean-up of a small website and deployed it to my home server. The server ran the pull, the build and the rollout, and Kubernetes reported success. None of the new code was running.
The server is a small computer at home running k3s, a lightweight Kubernetes. That month it also served the website, deployed by hand over SSH: pull the repo, build the image, push it to a local registry, restart the Deployment (Kubernetes’ record of which image to run). Every step after the pull reported that it had finished.
Nothing turned red, and the site was exactly what it had been. This is the case file: the suspects, the clue, the habit that most likely let it through, and the deploy script I’d use now. Two smaller cases from the same server close it out.
So how does a deploy fail at step one and succeed at step four?
the setupThe website went onto the server by hand: rsync the app over, build and push the image there, apply a manifest that existed only on the server.LessonAnything you copy or edit by hand on a server is something git will trip over later.
twistI committed the site, and the night before deploy day, its live manifest and the config for the tunnel that exposes the server, so a rebuilt server could be restored from git. The right call, but the next pull would now bring in files the server already had as untracked copies, and change one it had edited.LessonCommitting a file that sits untracked on a server breaks that server’s next pull.
incidentI committed a batch of small fixes and deployed. On the server, git pull refused to touch three paths and exited 1: two untracked copies from the hand deploy, and a tunnel config with its real ID filled in. The commands after it ran anyway: the build read the old files, and the rollout restarted the pods, pulled the same image and reported success.LessonA failed step only stops a chain if its exit status reaches the chain.
the clueThe new pods were running the image the old ones had. HEAD on the server, git’s name for the commit a checkout is on, had never moved.LessonA rollout proves your pods are ready, not that they’re new.
progressThree rules went into my deploy notes: pull as its own step, check HEAD is the commit I pushed, and check the image ID changed and the pods run the pushed digest. The three paths went into a timestamped backup folder, not the bin.LessonMove what blocks a pull aside. Don’t delete it: one of mine held the real tunnel ID.
closedThe site moved to Cloudflare Workers, because it went down whenever the server was off. The server’s main app was already deployed by CI, which builds the exact commit CI tested.LessonThe surest cure for a drifted checkout is not building from one.
The site’s Deployment used the :latest tag with imagePullPolicy: Always (pull the image again whenever a pod starts), and the recipe ended with kubectl rollout restart. A restart replaces every pod, Kubernetes’ running copy of the container, with a new one. Each new pod pulled :latest, which was the image the old pods already had, because the build had read the old files.
So Kubernetes did exactly what I asked: new pods, all ready, rollout complete. A rollout checks that pods are healthy, not that anything in them changed. And upstream of the build, HEAD on the server was still the old commit.
Every step had answered the question it was asked. None of them had been asked whether anything changed.
Wanted
for passing itself off as success
exit status 0
| tail -2, rm -f, a green rolloutReward: one changed image ID
Spoiler: it never lied. I asked it the wrong question.
git pull is two commands in a trench coat. First git fetch downloads the new commits and moves the remote-tracking branch (origin/main, say), git’s record of where the remote is. Then git merge moves your own branch to match and rewrites the files in your working tree.
The fetch worked. The merge found three paths it would have to overwrite: the site’s folder and manifest, untracked copies the new commits now tracked, and the tunnel config, a tracked file edited on the server. Git won’t overwrite work it can’t give back, so it printed an error, exited 1 and left HEAD where it was.
Rebuilt in two throwaway repos, with one edited file and two untracked ones in the way, it looks like this:
error: Your local changes to the following files would be overwritten by merge: deploy/tunnel.yml Please commit your changes or stash them before you merge. error: The following untracked working tree files would be overwritten by merge: deploy/site.yaml site/index.html Please move or remove them before you merge. Aborting
&& stops at a failure, if it ever hears about oneMy note from that afternoon says the pull ran inside a && chain whose later steps still succeeded. That doesn’t add up on its own: && runs the next command only if the last one exited 0, and in my reproduction a bare git pull --ff-only && stops cold. (--ff-only tells git to move forward only, never to merge.)
Something between the pull and the && must have hidden the failure. The likeliest suspect is a habit all over my earlier commands for that server: trimming each step’s output with | tail, like this one from July, when it still ran Docker Compose:
# run on the server over ssh cd ~/app && git pull 2>&1 | tail -2 && docker compose down -v 2>&1 | tail -2 && docker compose up -d --build 2>&1 | tail -4
Here’s the catch: a pipeline’s exit status is its last command’s. tail prints its lines and exits 0, so the pipeline exits 0 and && carries on. My reproduction, with one hand-made file in the way:
$ git pull --ff-only 2>&1 | tail -2 && echo "built $(cat app.txt)" Aborting Updating <old>..<new> built v1
The last line is Updating and two commit IDs. git writes that line to its normal output and the error to its error output; in a pipe, the normal output waits in a buffer until git exits, so it comes out last. Glance at the bottom of the output and it reads like progress.
“A manual deploy that silently builds a stale checkout is worse than a failed one. It looks green and ships nothing.”From my deploy notes, that afternoon
set -o pipefail would have caught it. The better fix is not to pipe the step that matters at all: pull on its own line, under set -e, and let it fail out loud.
My three rules from that afternoon become four checks, with one added at the start: is the checkout clean? Then: is HEAD the commit I pushed, is the image new, and are the pods running it? The old chain asked none of them, and each can stop the deploy and say why.
One caveat: an unchanged tree gives an unchanged image ID only with a warm build cache and a Dockerfile that builds the same way twice. The HEAD check is the guard; the image check is the alarm.
Save it as deploy.sh on your laptop, set the names at the top, and run it on the server through bash -s (bash reads the script from its input), so a zsh login shell (see Case 3) never runs it. Its comments name the four checks. It rolls out to Kubernetes; for Docker Compose, swap the roll-out and check 4 for the box after it, and delete docker push and the two digest lines, since there’s no registry.
#!/usr/bin/env bash
# deploy.sh: ship exactly one commit, or stop loudly.
# Run it from your laptop, right after git push:
# ssh server "bash -s -- $(git rev-parse HEAD)" \
# < deploy.sh
set -euo pipefail
want=${1:?usage: deploy.sh <commit you pushed>}
repo=~/app # checkout on the server
ctx=. # build context in the repo
img=localhost:5000/app/web # image name, no tag
ns=app # Kubernetes namespace
dep=web # Deployment name
ctr=web # its container
sel=app=web # label selector for its pods
die() { echo "DEPLOY STOPPED: $*" >&2; exit 1; }
cd "$repo"
# Check 1, a clean checkout: a drifted one stops here.
drift=$(git status --porcelain)
[ -z "$drift" ] || die "local changes in the way.
Move them aside (don't delete them), then rerun:
$drift"
# Pull on its own line. set -e makes a refusal fatal.
git pull --ff-only
# Check 2, HEAD: it must be the commit you pushed.
head=$(git rev-parse HEAD)
want=$(git rev-parse --verify "$want^{commit}")
[ "$head" = "$want" ] || die "HEAD is $head, not $want"
# Build, then check 3, a new image. Compare with the last
# deploy, not the last build, so a failed deploy can rerun.
tag=$img:${head:0:12}
old=$(docker image inspect -f '{{.Id}}' "$img:deployed" \
2>/dev/null || true)
docker build -t "$tag" -t "$img:latest" "$ctx"
new=$(docker image inspect -f '{{.Id}}' "$tag")
[ "$new" != "$old" ] || die "image did not change: $new"
docker push "$tag"
digest=$(docker image inspect "$tag" \
-f '{{range .RepoDigests}}{{println .}}{{end}}' |
sed -n "s|^$img@||p")
[ -n "$digest" ] || die "no registry digest for $tag"
# Roll out by tag, not by restart.
kubectl -n "$ns" set image "deployment/$dep" "$ctr=$tag"
kubectl -n "$ns" rollout status "deployment/$dep" \
--timeout=180s
# Check 4, every pod running the digest you pushed.
q='{.items[*].status.containerStatuses'
q+="[?(@.name==\"$ctr\")].imageID}"
stale=1
for try in $(seq 30); do
ids=$(kubectl -n "$ns" get pods -l "$sel" \
-o jsonpath="$q")
[ -n "$ids" ] || die "no pods match $sel in $ns"
stale=$(tr ' ' '\n' <<<"$ids" |
grep -vc "@$digest\$" || true)
[ "$stale" -eq 0 ] && break
sleep 2
done
[ "$stale" -eq 0 ] || die "$stale pod(s) not on $digest"
# Remember what's deployed, for the next run's check 3.
docker tag "$tag" "$img:deployed"
echo "DEPLOYED $want as $digest"git push && ssh server "bash -s -- $(git rev-parse HEAD)" < deploy.sh
docker compose up -d web
run=$(docker inspect -f '{{.Image}}' \
"$(docker compose ps -q web)")
[ "$run" = "$new" ] || die "web runs $run, not $new"I ran the script against throwaway repos, a local registry and a stand-in for kubectl that answers the way a cluster would. I haven’t run it against a real cluster for this post. Seven setups, seven answers:
| What I set up | What the script printed |
|---|---|
| An untracked folder and an edited file on the server | stopped local changes in the way, then M README.md and ?? deploy/ |
| A clean server and a new commit | deployed the commit, and the sha256: digest it pushed |
| A commit that only touched a file the image doesn’t include (here, the README) | stopped image did not change. That stop is on purpose: nothing new to ship |
| A push to a branch the server doesn’t track | stopped HEAD is one commit, not the one I passed in |
| Pods left on the old digest | stopped 1 pod(s) not on the pushed digest |
| A rerun after a rollout that failed | deployed check 3 compares with the last deploy, not the last build |
| A label selector that matches no pods | stopped no pods match |
Two weeks later I stopped trusting my notes and looked at the cluster. A nightly backup job had failed on every run since it was applied. The database dumps were fine; the upload was not:
CRITICAL: Failed to create file system for "<remote>:<path>": is a file not a directory
It reads like a path problem. The real cause: the Kubernetes Secret still held its template placeholder instead of a real storage endpoint, so rclone, the tool that copies the dumps to storage, spent four minutes failing to resolve a host that can’t exist, then reported a path error. Nothing in it said unconfigured.
The fix was a setup script that runs rclone lsd (list the target’s folders) against the target, from the same image the job uses, and won’t save a config the target hasn’t answered. Then a real backup, and a restore into a throwaway database, which passed. The line from my notes I keep coming back to: a scheduled job that has never succeeded looks exactly like one that succeeds nightly, from every angle except the artifact it was supposed to produce.
rm that deleted nothing, successfullyThe server’s /var/log is a small RAM disk, and old sysstat files had nearly filled it. I collected the old file names in a variable and deleted them over SSH. Trimmed to the lines that matter:
ssh root@server '…
for f in sa[0-9][0-9] sar[0-9][0-9]; do
… # keep the last 7 days
del="$del $f"
done
echo "deleting:$del"; rm -f -- $del …'
deleting: sa01 sa02 sa03 sa04 sa05 sa06 sa09 … sar29 sar30
No error, exit status 0, and every file still there. The difference was the shell. ssh host 'command' runs the command in the remote user’s login shell, and root’s login shell on that server is zsh. Unlike bash, zsh doesn’t split an unquoted variable into words, so rm got one argument: a single file name, dozens of names long, with spaces in it. No such file exists, and -f means a missing file isn’t an error.
Under bash, one file at a time, the same job worked. Earlier the same day the quirk had hit my Mac, also zsh, but loudly: curl rejected a whole unsplit option string as one unknown option. Same bug, once loud and once silent. The silent one is the dangerous one.
$ for sh in bash zsh; do $sh -c 'l="a b c"; printf "[%s]\n" $l'; done [a] [b] [c] [a b c]
Three cases, one pattern: each command reported on itself, and nobody checked the result. That afternoon in August I deployed and got a green light. My notes now make every hand deploy to that server pass four checks before it gets to say done: a clean checkout, the commit I pushed, a new image, and pods running it. Green is allowed back. It just has to show its receipts.
Case closedOn the server, compare HEAD with the remote. [behind 1] means the pull didn’t land, whatever the output said.
git status -sb git rev-parse HEAD origin/main
?? is a file git doesn’t track; M is a tracked file changed on the server.
git status --porcelain
I used a timestamped folder outside the repo. Inside git, a stash does the same job and keeps everything recoverable.
git stash push --include-untracked \ -m "pre-pull $(date +%F-%H%M)" git stash list
Check the ID before and after the build. Same ID, same code.
docker image inspect -f '{{.Id}}' <image>:latestCompare the sha256: part with the digest docker push printed. Kubernetes calls the registry digest imageID; it isn’t Docker’s image ID.
kubectl -n <ns> get pods -l <selector> \
-o jsonpath='{..imageID}'Start every script with set -euo pipefail, and don’t pipe the step that decides whether to carry on.
If it says zsh or fish, send scripts with bash -s or wrap them in bash -c.
ssh server 'echo $SHELL'
A run: step with no shell: set runs as bash -e, without pipefail, so npm test | tee test.log passes when the tests fail. Add shell: bash to the step, which runs it with -eo pipefail.
make runs each recipe line in its own shell and stops on a failure, but a line like pytest | tee out.log reports tee’s exit status. With SHELL := bash, start the line with set -o pipefail;. The neater .SHELLFLAGS needs GNU make 3.82 or later, and macOS ships 3.81.
cron’s only report is mailing whatever a job prints to its owner, and only if mail works on that machine. The same trap as Case 2: a scheduled job that fails every night can look exactly like one that works.
The first block recreates the half pull in two throwaway repos in a temporary folder. Run it in bash or zsh: it prints built v1, the old code, even though the pull failed.
cd "$(mktemp -d)" git init -q -b main origin && cd origin git config user.email you@example.com git config user.name you echo v1 > app.txt && git add . && git commit -qm one cd .. && git clone -q origin server echo hand-copy > server/site.yaml cd origin && echo v2 > app.txt && echo tracked > site.yaml git add . && git commit -qm two && cd ../server git pull --ff-only 2>&1 | tail -2 && echo "built $(cat app.txt)"
bash -c 'false | tail -1; echo $?; set -o pipefail; false | tail -1; echo $?'
git pull failed on the server, but git log origin/main -1 shows your new commit. Why?A pull is a fetch, then a merge. The fetch moved origin/main; the merge refused, so HEAD stayed where it was.
set -e, does git pull | tail -1 && build stop when the pull fails?No. set -e and && both see the pipeline’s exit status, which is tail’s 0. Add pipefail, or pull on its own line.
rm -f $files end with exit status 0 under zsh when nothing was deleted?zsh passes $files as one argument. No file has that name, and -f says a missing file isn’t an error.
|. Its status is the last command’s, unless pipefail is set.sha256:… address for one you pushed. Pods report the digest as imageID.git status marks it ??.