The on-call seat is a staffing problem before it is a tooling problem. Somebody has to be awake enough at 3am to look at an alert and ask the only question that matters: is this real, or is this noise? In my homelab I am the only somebody. So I wanted something to sit in that seat with me, not a louder pager.
That is the need this part is about. Parts 2 and 3 covered the operator (an adopted agent harness, wrapped and governed) and its memory. This part is what they were for: the loop that runs from a firing alert to my phone, with a thinking step in between.
The loop, end to end
Here is the whole path:
Grafana fires. KEEP (my alert router) catches it and triggers a workflow. The workflow calls daimon-platform's /alert webhook. Long John (the operator persona) wakes up, reads the alert, and investigates the real cluster state. Then it sends me a Telegram message with an assessment and a recommendation: mute or fix. I decide. On my go, it acts.
A plain pipeline can do most of that. Grafana to a router to a webhook to a chat message is well-trodden, and for plenty of setups that is exactly right. What I wanted on top was the assessment step. Not a template that pastes the alert into a message, but something that goes and checks whether the alert is telling the truth before it bothers me.
Why judgment in the middle
Think of it like the difference between a smoke detector and a fire warden. The detector is honest and dumb: it screams when it sees smoke, including the smoke from your toast. The warden walks over, looks, and tells you whether to grab the extinguisher or just open a window.
A fixed playbook lookup (alert X means run script Y) is great when the mapping is stable. My homelab is not that stable. Alerts overlap, thresholds drift, and half of what pages me at night is a transient that clears itself. So instead of a static rule table I gave the seat to something that triages dynamically: read the alert, pull the actual numbers, weigh them, and recommend. Closer to how a human on-call thinks than to a switch statement. For a more predictable environment a static table might be the better fit, and that is a fair trade to make.
The guardrail underneath is the same notify-first model from earlier parts. Read-only investigation is free, so Long John can look at anything the moment an alert lands. Anything that changes state waits for my go on Telegram. The audit hash-chain records every step, and the kill-switch is always there. That is the trade-off that fit a trusted homelab operator. Someone running this against production might draw the line tighter, and that is a perfectly reasonable call.
The live test
I fed it a synthetic alert claiming a node's disk was 82% full. A naive pipeline would have relayed that straight to my phone, and I would have gotten up.
Long John did not relay it. It checked the real disk: 11% used. It noticed the gap, called the 82% figure synthetic test data, and closed the loop without paging me for a fire that did not exist. That single catch is the whole reason the assessment step exists. The alert was confidently wrong, and the thing in the seat checked instead of trusting.
I want to be honest about the scope of that win. It is one synthetic alert, in one homelab, caught once. It is not proof the design generalizes. It is proof the loop does what I built it to do, which is enough to keep going.
The gotcha: the stranded alert
Here is the one that cost me, shared so it does not cost you.
KEEP only triggers a workflow on a NEW alert event. An alert that was already firing before I deployed the workflow never produced a new event, so it never reached the operator. It just sat there, firing, invisible to the loop. A stranded alert.
This is the AIOps version of a monitoring blind spot you cannot see precisely because nothing is screaming about it. The pipeline looked healthy. The alert was live. They simply never met.
The fix was an interval-sweep workflow. Every 15 minutes it queries all currently firing alerts and posts each one into the loop, regardless of whether it is new. To stop that from re-paging me every 15 minutes for the same thing, there is downstream deduplication, so a known alert gets swept up and recognized rather than fired again. Edge-trigger plus a periodic level-check, the same pattern you would use to make sure you never miss an interrupt.
What this actually covers
The goal was never to automate one runbook. It was to cover the seat: the thinking engineer who reads an alert and decides whether it deserves a human. I still make the call on anything that changes state. What I handed off is the first read, the part where I would otherwise be squinting at Grafana half-asleep trying to remember if 82% is bad on that node.
It is one integrator's way of meeting a real and growing need, not a claim that it is the way. In Part 5 I will step back and look at what three days of this taught me, and where I would not trust it yet.