From inbox spam to a phone buzz

My homelab used to email me when something broke. Good idea, right up until it became the problem.

I run kube-prometheus-stack on a small bare-metal Kubernetes cluster at home, with Alertmanager pointed at my inbox. The trouble was that a handful of alerts had been firing continuously for days: a parked workload, a couple of stale failed jobs, a daemonset that never fully settled. None were urgent. All of them re-emailed me every twelve hours, forever.

That is the worst possible outcome for a monitoring system: I stopped reading the emails. When the signal is buried under its own repeats, the channel is dead. So I tore it out and rebuilt how my homelab talks to me.

What I actually wanted

Three things, in order:

• A push to my phone only for things that genuinely matter (critical, not "warning").
• A dashboard for everything else, so the noisy-but-harmless stuff has a home I check on my own terms.
• Nothing leaving my own infrastructure. No third-party push service holding my alert text.

That last point ruled out the easy options. Telegram and Discord are zero-effort, but the alerts ride through someone else's servers. The self-hosted answer is ntfy: a tiny open-source push server with a phone app that I can run on my own cluster and still reach from anywhere.

The pipeline

The key decision was to not throw away the alerting rules I already had. kube-prometheus-stack ships a well-curated set of them, and Alertmanager already evaluates them. I only changed what happens at the end of the chain:

Prometheus rules -> Alertmanager -> ntfy bridge -> ntfy -> my phone
                                 \-> (warnings) -> Grafana, no push

Alertmanager keeps routing. Critical alerts go to a small adapter, the ntfy-alertmanager bridge, which turns Alertmanager's webhook into a cleanly formatted push. Everything below critical falls through to a null receiver: still recorded, just not on my phone. Grafana becomes the pane of glass for the full picture, which is exactly what a dashboard is for.

Mapping severity to push priority is a few lines:

labels {
    order "severity"
    severity "critical" {
        priority 5
        tags "rotating_light"
    }
}

The whole thing is declared in Git and reconciled by Argo CD, so the change is one reviewable commit, and one revert away from undo.

Gotcha #1: the phantom 500

While testing, I POSTed a hand-made alert payload at the bridge and got back HTTP 500. Cue twenty minutes of suspecting the bridge, the token, the network. The bridge was fine. My test was wrong.

Alertmanager's webhook payload is more than a list of alerts. It carries an envelope: a group key, the receiver name, common labels, a status field. My stub had the alerts array and little else, so the bridge choked on the missing fields. A real Alertmanager envelope returns 200 every time.

The lesson outlived the bug: when you test a webhook receiver, replicate the full shape the real sender produces, not a convenient stub. A minimal payload exercises a path the real system never takes, and you end up debugging a failure that does not exist.

Gotcha #2: reachable is not accessible

To get push on mobile data, ntfy has to be reachable from the public internet. The moment it was, the obvious worry showed up: if I can open the URL without logging in, cannot anyone?

Loading the URL and accessing the data are two different things. ntfy runs with deny-all access, so I checked what an anonymous visitor can actually do, and I checked it from inside the server rather than trusting the front door:

read my alerts topic  -> 403 Forbidden
publish to any topic  -> 403 Forbidden
create an account     -> disabled
health check          -> 200 (no data, by design)

Nothing readable, nothing writable, no signup. A login page rendering is not a leak; a login page that lets you in would be. There is one trap worth naming: ntfy defaults to open access. It is only locked because the config says so, which means the real risk is the config silently failing to load. That is why I verified at the source, not at the edge.

For good measure I turned the web UI off entirely, so the server now answers API calls only and an anonymous browser gets nothing to poke at. The phone app talks to the API, so it never noticed.

Where it landed

Now my phone buzzes for the things that would actually ruin my evening, the warnings wait quietly in Grafana until I go looking, and my inbox is back to being an inbox. The best part is not the push notification. It is that I trust the channel again, which is the only thing that makes a monitoring system worth keeping.