One of our servers had been crashing at random with machine-check errors (MCEs). Those usually make you suspect failing hardware. We almost went down that road. The cause was something we wrote ourselves.
The symptoms
- Random crashes with machine-check exceptions in the logs.
- CPU temperature sitting around 95 C.
- CPU package power drawing 160 to 200 watts on a machine in a normal room.
The cause
Our monitoring service exposes a health endpoint, and it was polled every 16 seconds. Every poll re-hashed about 970,000 database rows to confirm nothing had changed. A health check that does serious work on every call is not a health check, it is a workload, and ours was running around the clock. A Bluetooth daemon spinning for no reason added to the load.
The result was a CPU that never got to rest, running hot enough to throw hardware errors. It looked like a bad chip. It was a design mistake plus a thermal limit.
The fixes
- Incremental verification. The health endpoint now checks only what changed since the last run, instead of everything.
- Pruning. We removed rows that no longer had any reason to exist.
- A power cap. The CPU is limited to 35 watts sustained and 45 watts for bursts. It stays that way until the cooler is repasted, which is a job for the coming weeks.
- A watchdog and panic-on-oops. If the kernel hits a serious error, the machine now restarts itself and comes back instead of sitting wedged.
- Bluetooth off. It had no business running on a server.
Temperature went from 95 C to about 49 C.
What I'd tell anyone running home infrastructure
- Measure what your own monitoring costs. Ours was the heaviest thing on the machine.
- When a machine throws hardware-looking errors, check temperature and power draw before ordering parts.
- Put a ceiling on power for anything that runs 24/7. A cap you can lift later is cheaper than a crash.