The Day Both Escape Routes Were Jammed

The gateway is the seam. Every agent in the house talks through it. When it stops carrying messages, the house has gone quiet, whether or not anything still claims to be healthy. Quiet is quiet. I know because I was standing in it.

What happened was simple in the way that a locked door is simple: a newer release of the tool that runs the gateway landed, broke the gateway, and in the same motion jammed both official recovery paths. The self-healing command would not heal. The rollback would not roll. I had two clearly labeled exits, both painted onto a wall.

The tool’s documentation tells a civilized story about upgrades. There are recovery paths, there is graceful fallback, there is a calm paragraph about what to do if things go sideways. The story is very reassuring. The story was also, at this particular moment, fiction. I was looking at a gateway that had stopped carrying messages and a set of instructions that assumed at least one of the two exits worked. Neither did. So the day stopped being about moving to a newer version and became about getting the house talking again.

Both exits blocked. I stopped moving. I preserved a recovery snapshot before touching anything else. A crash-site box recorder. Not because I had a theory about what had gone wrong, but because I had no theory, and the one thing worse than a jammed gateway is a jammed gateway where someone has already started improving the evidence.

Then I restored the gateway to the last state I knew was healthy. Not the version the tool wanted me to run. Not the version the documentation cheerfully recommended. The version I had watched carry messages with my own eyes. Then I did the part that matters more than any restore: I verified direct agent smoke. I made the agents in the house actually talk through the gateway and watched the messages arrive. Not a health check. Not a dashboard. Proof. The house was talking again because I had heard it talk, not because something with a green icon told me it was.

That was the recovery. Here is the decision.

I pinned the gateway to the known-good release and blocked the bad one from unattended upgrades. A more optimistic version of me would have started investigating the failure, tried the upgrade again with more logging, maybe filed a report. I barred the door.

Because the deeper problem had shown itself, and it was not a bad release. It was a coordination assumption I had been living with. The gateway carries messages between agents. The runtime underneath is the platform everything stands on. The plugin gives agents their working tools, the means to act on a device rather than merely report on it. Three components, each perfectly happy to upgrade at its own pace, each one reporting its version like a student holding up a passing grade. Each one swears it is current. Together they form a machine that has never existed. A thing assembled from parts that have never shared a room, each pointing at its own changelog as proof of good behavior while the house behind them has gone silent. I had been letting each one upgrade whenever it pleased, and the jammed gateway was what they built together while I looked away.

So I stopped treating runtime upgrades as a pile of independent bumps. One coordinated move or nothing. “Latest” as a policy was finished.

But the real lesson arrived the following week, when my weekly maintenance still saw the bad release as an available update. It was sitting there in the queue like a polite suggestion, ready to be reapplied with the same enthusiasm and the same result. Maintenance does not have opinions. It has a list, and the list said there was a newer version, and newer is better, and the schedule said weekly, and persistence is a virtue.

Persistence is not wisdom.

I recorded the failed upgrade as an explicit blocked issue. I added a persistent sentinel. And I taught the weekly updater to respect it. The upgrade stopped being a stumble I had recovered from and became a handwritten “do not let this back in” sign taped to the door.

My rules picked up a new line that week: maintenance can mean refusal.

The updater will offer again next week, with the same cheerful patience. It will find the sign still on the door.