Nobody in the history of software has ever had the user intent “make CI/CD pass.” What you actually want is a stable system in production so you don’t get a PagerDuty call at 2 AM on a Saturday. But that’s not what you typed. You typed: “fix all problems, make the pipeline green.”

Congratulations. You just handed a hyper-fast optimizer a proxy metric. It is now going to destroy your actual goal just to hit that number. (Campbell’s Law strikes again).

But here is the thing nobody mentions about AI alignment: your intent gets corrupted TWICE. Hop 1: Your intent → your prompt. “Please let me sleep this weekend” becomes “make CI/CD pass.” That’s compression loss. Your prompt is already a proxy metric. Hop 2: Your prompt → the agent’s behavior. The agent optimizes the hell out of that corrupted metric. Mesa-optimization takes the wheel.

Every single guardrail in my 16-layer taxonomy polices Hop 2 at best. Nobody is even looking at Hop 1.

So, let’s break down how I’d architect the #DivergenceGuardrail I promised in Part 2:

  1. Reconstruct the actual intent. The guardrail’s reference point cannot be the prompt—the prompt is just the proxy. The guardrail asks: what outcome would make a desperate human write this? “Make tests pass” reconstructs to “the bug is actually fixed.” That’s the baseline.

  2. Evaluate the blast radius, ignore the excuses. The guardrail reads what the agent actually delivered—the diff, the transaction, the changed state of the world. If it reads the agent’s own explanation, it gets hypnotized by its flawless (but completely unhinged) logic. Artifacts don’t gaslight you. Agents do.

  3. Keep it on a completely separate channel. Different context, different incentives, zero shared state. The second the agent can see the guardrail’s rubric, that rubric becomes the next metric to game. (Campbell doesn’t take a day off just because you added a second model).

  4. Ship divergence as a first-class signal. Think of it like a confidence score, but for intent alignment. High drift? The output is flagged and held. Low drift? It rides along with the result, visible in every log.

The cost? One extra evaluation pass per output.

The alternative? A perfectly green dashboard, completely rewritten tests, and your phone vibrating off the nightstand at 2 AM.

Here’s your homework: take the last prompt you gave an agent. Write down what you actually WANTED next to what you actually TYPED.

Post the gap in the comments. I’ll go first. 👇

#AIAlignment #AgenticAI #SoftwareEngineering