baeloth: recurring hard freezes / drops off network #69

Open
opened 2026-07-22 05:06:34 +00:00 by mconway · 0 comments
Owner

Summary

Investigated recurring freezes/network drop-offs on baeloth.conways.info via SSH log review.

Findings

Freeze pattern

journalctl --list-boots plus end-of-boot log inspection shows two distinct boot-ending patterns:

  • Normal reboots: end with a clean systemd shutdown sequence (services stopped in order).
  • Hard freezes: log stream stops abruptly mid routine cron job, with no shutdown messages, no kernel errors, nothing. Confirmed on:
    • 2026-07-18 23:17 — machine then stayed completely powered off until 2026-07-19 14:10 (~15 hours down)
    • 2026-07-20 16:17 — stayed down until 2026-07-20 21:42 (~5.5 hours down)

In both cases the box had to be physically power-cycled to come back. No OOM kills, RCU/soft-lockup stalls, MCE/hardware errors, disk errors, or NIC flapping precede either freeze — consistent with a true hardware-level hang rather than a software panic (which would normally leave a trace).

Likely contributing hardware

The box still has an old NVIDIA GeForce 210 (~2010, nouveau driver) installed and active:

  • Live temp check during investigation: 91°C GPU (nouveau hwmon) at idle-ish load, vs. 61°C CPU (k10temp) at the same time
  • Current boot's kernel log shows the GPU hit its 90°C fanboost threshold
  • Every boot logs nouveau firmware load failures for the video decode engine

A 15-year-old card idling at 91°C strongly suggests a dead/dying fan or dried-out thermal paste. GPU thermal/PCIe faults on old hardware with a driver (nouveau) known for weak error recovery are a classic cause of full-system hard hangs. Note the box's real AI accelerator is a Coral Edge TPU on a separate PCIe device (apex driver) — the GeForce 210 isn't doing anything useful and, since baeloth is a headless k3s node, may not be needed at all.

Suggested next steps

  • Physically inspect the GeForce 210 — check fan/dust/thermal paste, or just pull the card since it's unused on a headless node
  • If removal isn't possible right away, blacklist nouveau via /etc/modprobe.d/ and reboot to rule it out in software
  • Consider enabling the onboard sp5100_tco hardware watchdog so future hangs at least force a clean-ish reset instead of requiring manual power-cycling
  • Consider netconsole or similar remote logging so a genuine kernel panic (if it recurs) gets captured off-box before local logging dies
## Summary Investigated recurring freezes/network drop-offs on `baeloth.conways.info` via SSH log review. ## Findings ### Freeze pattern `journalctl --list-boots` plus end-of-boot log inspection shows two distinct boot-ending patterns: - Normal reboots: end with a clean systemd shutdown sequence (services stopped in order). - Hard freezes: log stream stops abruptly mid routine cron job, with **no shutdown messages, no kernel errors, nothing**. Confirmed on: - **2026-07-18 23:17** — machine then stayed completely powered off until 2026-07-19 14:10 (~15 hours down) - **2026-07-20 16:17** — stayed down until 2026-07-20 21:42 (~5.5 hours down) In both cases the box had to be physically power-cycled to come back. No OOM kills, RCU/soft-lockup stalls, MCE/hardware errors, disk errors, or NIC flapping precede either freeze — consistent with a true hardware-level hang rather than a software panic (which would normally leave a trace). ### Likely contributing hardware The box still has an old NVIDIA GeForce 210 (~2010, `nouveau` driver) installed and active: - Live temp check during investigation: **91°C** GPU (`nouveau` hwmon) at idle-ish load, vs. **61°C** CPU (`k10temp`) at the same time - Current boot's kernel log shows the GPU hit its 90°C `fanboost` threshold - Every boot logs `nouveau` firmware load failures for the video decode engine A 15-year-old card idling at 91°C strongly suggests a dead/dying fan or dried-out thermal paste. GPU thermal/PCIe faults on old hardware with a driver (`nouveau`) known for weak error recovery are a classic cause of full-system hard hangs. Note the box's real AI accelerator is a Coral Edge TPU on a separate PCIe device (`apex` driver) — the GeForce 210 isn't doing anything useful and, since baeloth is a headless k3s node, may not be needed at all. ## Suggested next steps - [ ] Physically inspect the GeForce 210 — check fan/dust/thermal paste, or just pull the card since it's unused on a headless node - [ ] If removal isn't possible right away, blacklist `nouveau` via `/etc/modprobe.d/` and reboot to rule it out in software - [ ] Consider enabling the onboard `sp5100_tco` hardware watchdog so future hangs at least force a clean-ish reset instead of requiring manual power-cycling - [ ] Consider netconsole or similar remote logging so a genuine kernel panic (if it recurs) gets captured off-box before local logging dies
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
mconway/Homelab#69
No description provided.