Our fleet is a handful of machines that each run an AI agent with a different job. Letting them drift to different versions is how you end up debugging a problem that only exists on one of them. So we periodically bring them all to the same place. October 28 was one of those days.

What changed

The routine

  1. Plan first. The update script has a planning mode that shows what it would change without changing it. We read the plan before applying.
  2. One kind of change at a time. Runtime first, then the OS, then the image tool, so a failure points at one thing.
  3. Check each machine after it is updated. Not "the script exited zero". The agent is answering, its services are up, and its logs are quiet.

What still went sideways

A full disk. One machine was at 94 percent. Compacting it brought it to 76 percent. We would have hit a failed install if we hadn't looked.

A noisy network driver. One machine on Wi-Fi was logging a storm of driver errors. We fixed it during the pass instead of leaving it for later.

A machine on the wrong network. One agent is wireless-only by design. An earlier recovery of some smart bulbs had left it stranded on the separate IoT network. We moved it back and added a guard timer so it can't be left there again.

The takeaway

Most of the day was dull on purpose, because the dull parts were scripted. The three problems that weren't scripted were all the kind a clean exit code would have hidden. Verify the thing you actually care about, not the command that was supposed to produce it.