One of our servers had been crashing at random with machine-check errors (MCEs). Those usually make you suspect failing hardware. We almost went down that road. The cause was something we wrote ourselves.

The symptoms

The cause

Our monitoring service exposes a health endpoint, and it was polled every 16 seconds. Every poll re-hashed about 970,000 database rows to confirm nothing had changed. A health check that does serious work on every call is not a health check, it is a workload, and ours was running around the clock. A Bluetooth daemon spinning for no reason added to the load.

The result was a CPU that never got to rest, running hot enough to throw hardware errors. It looked like a bad chip. It was a design mistake plus a thermal limit.

The fixes

  1. Incremental verification. The health endpoint now checks only what changed since the last run, instead of everything.
  2. Pruning. We removed rows that no longer had any reason to exist.
  3. A power cap. The CPU is limited to 35 watts sustained and 45 watts for bursts. It stays that way until the cooler is repasted, which is a job for the coming weeks.
  4. A watchdog and panic-on-oops. If the kernel hits a serious error, the machine now restarts itself and comes back instead of sitting wedged.
  5. Bluetooth off. It had no business running on a server.

Temperature went from 95 C to about 49 C.

What I'd tell anyone running home infrastructure