By Part 4 of this series, my AI engineer could take an alert, think about it, and message me with a recommendation. By Part 5 it had survived three days of real incidents. That felt like operations. It wasn't, quite.
Here is the thing I had to admit to myself: an AI that reacts to every alert is a very fast pair of hands, not an operations function. Anyone who has run a service desk knows the difference. Alerts are noise. There is a discipline that turns that noise into work: it folds twenty pages from one dead node into a single thing, assigns it to someone, tracks its status, and keeps a record after the dust settles. It has a name. Incident management. ITSM, if you like acronyms.
So Part 6 is the part where I stopped treating my AI like a smart alarm and started running it like an engineer on a rota, with a queue, a desk, and a paper trail.
The wrong turn I talked myself out of
My first instinct was to make the AI do everything: ingest the alerts, decide what counts as an incident, work it, and remember it. One brain to rule it all. I talked myself out of that, and I am glad I did. It is the same mistake I have watched plenty of teams make with humans, asking the engineer to also be the ticketing system. You end up with neither done well.
So I split the job. There is a layer whose only purpose is incident management: grouping, deduplication, assignment, status, the system of record. I did not write it. I reached for an open-source tool built for exactly that (KEEP, in my case, though there are others, and I am not here to tell you which to pick). And there is the engineer, the AI operator I have been calling Long John through this series, whose job is to work incidents, not to run the queue.
That distinction sounds obvious written down. It took a wrong turn for me to actually believe it. The integrator's job, most days, is deciding what not to build because someone already built it better, and then wiring the pieces so each one does a single thing well.
The orchestration
Once the roles were clean, the loop almost designed itself.
- Alerts arrive from everywhere: the cluster, the firewall, the NAS, the apps.
- The incident layer correlates them. Twenty alerts from one failing node become one incident, not twenty pages. That alone is most of the value.
- The incident is born already assigned to the AI engineer.
- The engineer is handed the incident, looks at the real state of the system rather than what the alert claims, tells me what it sees and what it intends before it touches anything, and then, with my nod, does the work.
- And it writes its work back into the incident: what it found, what it did, how it closed.
That last step is what makes it ITSM and not just automation. The desk holds the record, not the engineer's memory and not a buried chat log. The next person to look, human or AI, reads the incident and sees the whole story. Six months from now, "why did the NAS reboot in June" has an answer that is not trapped in my head.
The part that humbled me
I will be honest about where it got awkward. I had given the AI the role, the protocol, the instruction to work incidents on the desk, and then it kept asking me how to actually reach the thing. I had told it what to do without giving it a usable key or a reachable address. It was like handing someone a job description for a system they have no login to.
That is a small, very human mistake, and it is worth saying out loud, because it is the real lesson of this whole journey. Giving something a role means giving it the actual tools and reachable access, not just a good-sounding instruction. A protocol that promises access the environment never provides is just a frustrated engineer. Once I fixed the plumbing (a key with the right permissions, the right address) it simply worked. The discipline was sound; I had skipped a wire.
Why bother, for a homelab
Because the moment you let an AI do real operations (change configs, restart services, touch the things that matter) you need the same scaffolding you would insist on for a human. A queue so nothing gets dropped. A single record so there is accountability. A lifecycle so "open" and "resolved" actually mean something. An audit trail so you can answer "what happened, and who decided what." Without that you do not have AIOps. You have a clever script with root, and that should make you a little nervous.
Incident management is the unglamorous part, and it is exactly the part that makes AI operations auditable instead of magical. The market is going to want AI in the operations loop, and that need is real and growing. But it will want it accountable. The incident desk is how you get there.
Where it leaves me
The system is not finished. The correlation rules need tuning, the attribution needs polishing, and there is a long list of later. And I want to be clear that this is one shape, not the shape. You could wire this a dozen ways with a dozen tools and be just as right.
But imposing the incident-management discipline is what turned a pile of working parts (the brain, the body, the rules, the judgment from the earlier parts) into something that behaves like an operations function instead of a demo.
I did not build an AI engineer and point it at alerts. I gave it a desk, a queue, and the habit of writing things down. That is the part that made it feel less like a trick and more like a hire.