Docker Compose Upgrade: A Forgotten Cron Job Caused an Outage

The upgrade went fine. The outage happened the next night, at 02:00, and the cause was a line in /etc/crontab that had been sitting there for five years.

This is a blameless write-up of upgrading Docker Engine on two Linux hosts (staging and production), moving from the old Python docker-compose v1 to the Compose plugin, and the incident that followed. The services are an API, a WebSocket server, workers, MySQL and Redis, all in one compose project.

What You'll Learn

✅ What a Docker Engine upgrade actually does to running containers
✅ The Compose v1 to v2 differences that bite in production
✅ Why links and container names turn a partial recreate into an outage
✅ The incident timeline and what the cron job had to do with it
✅ A pre-upgrade checklist, and a post-upgrade hunt for scheduled jobs

1. What a Docker Upgrade Really Does

Upgrading the docker-ce package restarts the Docker daemon. When the daemon stops, every container stops, and when it comes back only containers with a restart policy start again.

Before the upgrade we listed every container's restart policy:

for c in $(docker ps -aq); do
  docker inspect "$c" \
    --format '{{.Name}}  restart={{.HostConfig.RestartPolicy.Name}}  {{.State.Status}}'
done | sort

Several containers showed restart=no: a pair of workers on one host, and two unrelated connector services living on the same machine on the other. They were started by hand or by an older compose file years ago, and nobody remembered. Left alone, the upgrade would have silently killed them.

docker update fixes a running container without recreating it:

docker update --restart always <container> <container>

Then fix the compose file, so the policy survives the next recreate. Fix both, or the problem returns on the next deploy.

Also check what else lives on the host. A shared machine means your upgrade is somebody else's outage.

Use the restart you already have

A restart of the daemon is unavoidable, so we used it. One of the hosts had "debug": true in daemon.json, which wrote about 4,700 lines a minute to the journal, including container environment variables, passwords among them. We fixed it in the same window and added log rotation:

{
  "debug": false,
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "live-restore": true
}

Validate the file before restarting. A broken daemon.json means Docker doesn't start:

sudo dockerd --validate --config-file /etc/docker/daemon.json

live-restore keeps containers running through future daemon restarts. Don't count on it for the upgrade that introduces it.

2. Compose v1 to v2: The Differences That Bite

The old docker-compose (1.24, a Python script) and the docker compose plugin are different programs. Three differences mattered.

The old binary can't talk to the new engine

docker-compose 1.24 speaks an old API version. A current engine rejects it, so after the engine upgrade every existing deploy script that calls docker-compose is broken. On one host it was already broken before we started. Our fix was to point the old name at the plugin:

sudo ln -sf /usr/libexec/docker/cli-plugins/docker-compose /usr/local/bin/docker-compose

That keeps the old scripts working. It also means they now run with v2 semantics, which brings us to the next two items.

Container names changed

v1 names containers project_service_1. v2 names them project-service-1. Anything that refers to a container by name (docker exec, monitoring, scripts) needs updating. More importantly, the same service may exist under two naming schemes for a while.

Environment files are interpolated

Compose v2 expands $ in env_file values; v1 did not. A database root password containing a $ was silently truncated when the container was recreated. The fix is to escape it as $$ in the env file.

The tricky part is verification: docker compose config prints the already-escaped form, so it proves nothing. The reliable check is to run a one-off container and print the real value (or at least its length):

docker compose run --rm --no-deps -T --entrypoint printenv mysql MYSQL_ROOT_PASSWORD </dev/null | wc -c

3. Why a Partial Recreate Is an Outage

Our compose file still used the legacy links: option. When a service is created under compose v1, the link to mysql resolves through the container's old name (..._mysql_1). Recreate the mysql container under v2 and that old name no longer exists.

Containers that outlived the recreate keep their old link configuration. Docker's embedded DNS can no longer resolve mysql for them:

Unknown server host 'mysql'

docker restart doesn't fix this, because the link is part of the container's configuration, not its process. Only recreating the container does.

So the rule is: recreate every service that depends on the thing you recreated, together.

docker compose up -d --no-deps --force-recreate restapi websocket worker worker_manager

(--no-deps stops Compose from also recreating or starting the services' dependencies, so you decide exactly what restarts.)

4. The Incident

Day 1, around 03:40. Docker upgraded on production. Services came back. We noticed the old docker-compose binary no longer worked against the new engine and symlinked it to the plugin.

Day 2, 02:00. An API outage. No deploy was running and nobody was logged in.

The facts, reconstructed from logs and docker ps:

• mysql, redis and the workers had been recreated under the new v2 names at about 02:00
• The REST API and WebSocket containers had not. They were still the old ones, with links to the removed v1 container names
• The REST API threw Can't reconnect until invalid transaction is rolled back, then worker timeouts, and later Unknown server host

Something had run docker-compose up -d at 02:00 and recreated only part of the stack.

The culprit was in /etc/crontab:

0 2 * * * root sudo -i sh /data/<app>/restart_workers.sh

A nightly script that restarts the workers, added in 2021 and not documented anywhere. For five years it ran the old docker-compose up -d on the worker services, and it was harmless. The night after we symlinked docker-compose to v2, the same script recreated mysql, redis and the workers under new v2 names (most likely because v2 saw the v1-created containers as out of date), while the API and WebSocket containers kept pointing at ghosts.

We fixed it at about 06:30 by force-recreating the API and WebSocket containers together. The smoke test passed on the same image that had been running before.

5. The Part That Should Make You Check Your Own Hosts

The staging host had the same kind of line, added a couple of weeks earlier than production's. But its script name was misspelled: restart_workers.h instead of restart_workers.sh. Cron had been trying to run a file that didn't exist, every night, for five years, and nobody noticed because nothing was broken.

The two lines show the same lesson from opposite sides:

• The working cron job was a hidden dependency that changed behavior when we changed the tooling underneath it.
• The broken cron job was dead code: it would have caused the same outage if someone had fixed the typo.

6. What We Changed

✅ Commented out both cron lines (a restart "just in case" isn't a fix, and nobody knew why it existed)
✅ Added --no-deps to the up -d calls in the restart script, so it can never recreate the database
✅ Made the deploy script recreate the whole dependent group when it touches mysql or redis, and check for the Unknown server host symptom
✅ Documented the restart-policy audit and the $$ check in the upgrade runbook
✅ Disabled debug logging and added log rotation

A Checklist for Your Next Docker Upgrade

Before

• List restart policies for every container on the host, including other teams' (docker inspect).
• Fix them with docker update and in the compose files.
• Record the current versions (docker version, apt-cache policy) so you know what to roll back to.
• Validate daemon.json with dockerd --validate.
• Search for anything that calls Docker without you: crontab -l, /etc/crontab, /etc/cron.*, systemd timers (systemctl list-timers), and deploy scripts.

During

• Expect every container to restart. Plan the window.
• Upgrade engine, CLI, containerd and the compose plugin together.

After

• Check that the old docker-compose name works, or deliberately removes it.
• Check container names, links and any script that uses a container name.
• Verify env values with a one-off container, not compose config.
• Recreate dependent services as a group.
• Wait for one full day before declaring victory: nightly jobs, log rotation and scheduled restarts only show up on their own schedule.

Summary

A Docker upgrade is not just a package update: it restarts everything, changes what docker-compose means, and exposes every scheduled job that ever touched Docker. We got the planned work right and still had an outage, because the riskiest component was one we didn't know existed.

The incident wasn't caused by the upgrade or by the cron job alone. It was caused by an undocumented job meeting a changed tool.

Search for the jobs nobody remembers. Better to find them on a Tuesday afternoon than at 02:00.

Related Posts

• Migrating from Flask to FastAPI
• From RQ Workers to an Asyncio Engine
• Upgrading Django 5.2 to 6.0