Who’s Watching the Watchers: Scheduled Self-Reporting and an Independent Watchdog

Who’s Watching the Watchers: Scheduled Self-Reporting and an Independent Watchdog

The gap self-reporting alone can’t close

Once I had multiple service agents running, the natural next step was having them check on themselves regularly instead of me remembering to ask each one individually. But building that revealed a real blind spot: a scheduled check only runs if the machine it’s scheduled on is actually up. If the whole primary machine goes down, there’s nothing left to run the check and nothing gets reported — silence, not an alert, which is the worst possible failure mode for a monitoring system. Scheduled Self-Reporting is really two systems that close that gap together.

System one: a director that fans out and rolls up

A coordinator agent wakes on a schedule and fans out to each service agent (batched if there are many, to avoid hammering everything at once). Each agent runs a real domain-specific check — not a generic ping, but “is the library actually loading,” “is the container actually healthy” — attempts its own recovery if it can, logs the result, and reports back. The director collects everything and sends exactly one alert if something couldn’t recover on its own, never one alert per agent for the same root cause. A separate, slower-cadence job runs the same fan-out shape for a different question: does each service have an update available, apply patch/minor versions automatically, and escalate anything requiring a human decision (a major version bump) into a full written report instead of a one-line status.

System two: the independent watchdog, on genuinely separate hardware

This is the piece that actually matters most, and it’s deliberately not fancy: an old laptop, wiped and put on Linux, physically and electrically separate from the main server — different power circuit, ideally a different network path. It pings the primary machine every few minutes, tiered from a basic reachability check up to a systemctl is-active precision check where SSH access allows it. On a confirmed failure (after retries — transient blips shouldn’t trigger a false alarm and train me to ignore real ones), it alerts immediately, attempts a tiered fix (restart the specific service first, escalate to power-cycling only if that fails), polls for recovery within a bounded window, and sends a second alert either way.

The test that actually matters isn’t “does the watchdog catch a stopped service” — it’s pulling the network cable on the main machine and confirming the watchdog survives and alerts anyway. That’s the only thing that proves the “independent” part is real and not just two processes on the same box.

A credential problem with no clean answer, and the honest tradeoff I made

The watchdog’s whole reason for existing is surviving the primary machine being down — which means it can’t depend on that primary machine’s password manager for its own bot token, the way every other agent on the Tower does. That rule (route every credential through the central vault) directly conflicts with the reason this device exists in the first place. I didn’t find a way around that tension, because there isn’t one — instead I used systemd-creds (built into modern systemd, no extra software) to encrypt the token at rest with a key only the local machine holds, loaded automatically by the watchdog’s own systemd service. It’s not zero-plaintext-anywhere; it’s the honest middle ground once the primary design constraint (survive the primary machine’s death) rules out the usual answer.

Also load-bearing: a maintenance-mode flag that suppresses the whole scan for a set window during planned work, auto-expiring so a forgotten flag can’t silently disable monitoring forever — and other scripts can flip it on themselves (an update script starting maintenance mode right before it begins work) rather than relying on me to remember.

What it demonstrates

This project is about taking “monitoring” seriously enough to notice its own single point of failure, and then actually building the fix rather than accepting the gap. The credential tradeoff on the watchdog is a good example of engineering judgment under a real constraint — not blindly following a security rule that was written for a different context, and not ignoring it either, but finding the actual best answer once the rule’s own assumption (the primary machine is reachable) doesn’t hold.

Part of the Homelab AI Blueprints series — see all nine blueprints for building and operating a self-hosted, AI-agent-run home server.