baeloth: recurring hard freezes / drops off network #69
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Investigated recurring freezes/network drop-offs on
baeloth.conways.infovia SSH log review.Findings
Freeze pattern
journalctl --list-bootsplus end-of-boot log inspection shows two distinct boot-ending patterns:In both cases the box had to be physically power-cycled to come back. No OOM kills, RCU/soft-lockup stalls, MCE/hardware errors, disk errors, or NIC flapping precede either freeze — consistent with a true hardware-level hang rather than a software panic (which would normally leave a trace).
Likely contributing hardware
The box still has an old NVIDIA GeForce 210 (~2010,
nouveaudriver) installed and active:nouveauhwmon) at idle-ish load, vs. 61°C CPU (k10temp) at the same timefanboostthresholdnouveaufirmware load failures for the video decode engineA 15-year-old card idling at 91°C strongly suggests a dead/dying fan or dried-out thermal paste. GPU thermal/PCIe faults on old hardware with a driver (
nouveau) known for weak error recovery are a classic cause of full-system hard hangs. Note the box's real AI accelerator is a Coral Edge TPU on a separate PCIe device (apexdriver) — the GeForce 210 isn't doing anything useful and, since baeloth is a headless k3s node, may not be needed at all.Suggested next steps
nouveauvia/etc/modprobe.d/and reboot to rule it out in softwaresp5100_tcohardware watchdog so future hangs at least force a clean-ish reset instead of requiring manual power-cycling