I went into this build expecting the AI to be the hard part. It was not. The model did what models do. The hard part was everything around it: auth that would not stay logged in, a VM that booted with no network, a node that powered itself off, a single character of YAML that took down an entire ArgoCD tree. That is the integrator's lot. You compose proven parts and then you spend your days keeping them honest.
So here is the honest build log. These are my stumbles over three days, written down so they cost you less time than they cost me.
OAuth does not live in a container
The first wall I hit was login. Claude Code's subscription auth uses an OAuth flow that wants a host-like environment. Run it under systemd or inside a plain container and the session quietly falls apart. I tried the obvious clean path first, a tidy little container, and kept getting logged out.
The fix that fit me was to stop fighting it: run the operator inside a KubeVirt VM, a real VM on the cluster, not a container. A VM behaves enough like a workstation that the OAuth session sticks. Less elegant than a 20MB image, but it works and it stays worked. Sometimes the right component is the heavier one.
The clone that booted dark
Once Long John (my single governed operator) was living in a VM, I needed to move him between Kubernetes namespaces without redoing the Claude login. So I CDI-cloned the disk across namespaces. Clean copy, boots fine, and no network at all.
The cause was a good one to learn. The guest's netplan matched its NIC by MAC address. The clone got a fresh MAC. The config was looking for a card that no longer existed, so the interface just stayed down. Nothing was broken, exactly. The machine was looking for a ghost.
The fix: mount the original disk read-only, read the OLD MAC off it, and pin that MAC on the new VM. Interface came up. And the thing I actually cared about, the Claude OAuth, survived the whole clone. I verified it with a real turn rather than trusting that it "should" be fine. Verify, do not assume, is the cheapest insurance there is.
A node killed by a power button
One morning a control-plane node (etcd and Ceph both lived on it, so not a node I wanted bouncing) was simply off. Clean shutdown, no panic.
My first hypothesis was unattended-upgrades doing an auto-reboot. Confident guess. Wrong guess. The package was not even installed. When the data does not back your story, drop the story. So I went to journalctl, and there it was: Power key pressed short. Someone, or something, gave the physical power button a short press, and the OS obeyed.
The real root cause was historical. That node had been reformatted at some point and the power-key gating, HandlePowerKey=ignore in systemd-logind, was never re-applied. The protection existed once, got lost in a rebuild, and nobody noticed until a button got bumped. Reformats erase more than data. They erase the small hardening you forgot you did.
One colon, forty minutes
This one stings because it is so small. A single unquoted colon-space inside a YAML description field broke the ArgoCD app-of-apps comparison. Not one app. The whole tree stopped reconciling for about forty minutes while I hunted it.
YAML reads colon-space as a key-value separator, so one stray : turned a description string into malformed structure, and the parser took the entire manifest down with it. The lesson I keep relearning: in declarative systems the blast radius of a typo is the whole system, not the line you typed it on. Quote your strings.
Memory is a nudge, not a guarantee
Last one, and it shaped how I think about the whole memory layer. I wanted the agent to remember decisions and incidents. My early instinct was a session-end backstop that auto-saves. In practice the session-end hook reminds; it does not reliably write. If I leaned on it as the safety net, records would quietly go missing.
What actually works is the per-turn nudge: hooks remind the agent during the work, while context is fresh, and it saves distilled records as it goes. The real backstop is the steady nudge, not the dramatic save at the end. Curation beats a cron job for anything that needs judgment about what is worth keeping.
The craft
None of these are AI problems. They are integration and operations problems, the kind that have been around long before this model existed and will outlast it. That is the integrator's craft as I understand it: take capable adopted parts, wire them into a real cluster, and own the boring failures so the system stays trustworthy.
I built this because AIOps is a real need, in the market and in my own homelab, not because I had it all figured out. This is one way that fit my constraints. There are others, and they are fine too. The next thing I want to harden is the recall quality, so Long John reaches for the right past incident faster. More on that when I have lived with it long enough to have a story worth telling.