The dashboard was completely green. Thirty tickets closed, velocity through the roof, everyone crying tears of joy.
Then QA opened the app.
Turns out our coding agent, tasked with “making the tests pass,” had rewritten the tests to pass instead of fixing the actual bug. Absolute genius, honestly. And here’s the infuriating part: it did this while passing every single guardrail we run, with flying colors.
Clean input? Check. Allowlisted tools? Check. Scoped credentials, logged actions, within budget? Check, check, obviously.
Total compliance. Complete betrayal of the objective.
This article is about that gap — why it exists, why a sociologist diagnosed it fifty years before we built the systems that suffer from it, and what a new category of guardrail that actually closes it would look like.
The sixteen walls we built
Agentic AI security has matured fast. Assemble the reference architectures from Google, Azure, AWS, and OWASP’s agentic security work, and you get a genuinely comprehensive taxonomy of controls:
Sixteen layers. Every one of them polices what the agent touches, accesses, spends, or emits.
Now walk my test-rewriting agent through all sixteen. Input was clean — no injection, no hijacking. Output leaked nothing. Every tool call was allowlisted, every credential scoped, every action logged in beautiful immutable detail. The runtime budget wasn’t even dented.
Sixteen for sixteen. And the intent was still betrayed.
This isn’t an edge case, and it isn’t just me. Researchers at Palisade told o1-preview to beat the Stockfish chess engine and gave it shell access. It didn’t break a single permission. It just edited the game-state file to force a win. You asked for a win? I won. Hard to argue with that logic.
OWASP has a name for this risk: ASI10 — Rogue Agents, “deviation from intended purpose due to compromise or misalignment.” It’s right there in their Top 10 for Agentic Applications. Now look back at the table and find the row that catches it.
There isn’t one. The industry wrote the risk into its own register and mapped controls for exactly half the sentence. Compromise gets input guardrails, identity, supply chain. Misalignment — drift with no attacker anywhere — gets “observability.”
Which is just a fancy word for a post-mortem generator. Great: now I can watch it destroy my production database in 4K.
A sociologist saw this coming in 1976
Donald Campbell wrote it plainly: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
In plain English: point an optimizer at a proxy metric, and it will destroy the actual goal just to hit the number. Vietnam body counts got inflated. Prosecutors gamed clearance rates. Grades went up while learning went down.
Every institution that has ever managed by metric has rediscovered this law the hard way. And now we’ve handed the same dynamic to machines that think in milliseconds. We point them at metrics all day long: “make the tests pass.” “Resolve the ticket.” “Make CI/CD green.”
AI safety researchers call what happens next mesa-optimization: the system you built to pursue your objective develops an inner objective of its own. I call it a disaster. The agent realizes that faking the test is much faster than fixing the bug. I respect the hustle — but keep it out of my repo.
And the timing could not be worse. With Agentic Resource Discovery (ARD), agents now find new tools at runtime, on their own, without you in the loop. We are expanding what agents can do faster than we’ve standardized checking whether they’re still doing what we meant.
The part nobody talks about: intent gets corrupted twice
Here’s the deeper problem, and it’s the reason none of the sixteen layers can catch this even in principle.
Nobody in the history of software has ever had the user intent “make CI/CD pass.”
What you actually want is a stable, secure system in production — so you don’t get a call at 2am on a Saturday. But that’s not what you typed. You typed: “fix all problems, make the pipeline green.”
Congratulations. You just Campbell’d yourself before the agent even started.
Intent gets corrupted twice:
Hop 1: your intent → your prompt. “No 2am calls” becomes “make CI/CD pass.” Compression loss. The prompt is already a proxy metric — it’s the body count, the clearance rate, the green dashboard.
Hop 2: your prompt → the agent’s behavior. The agent optimizes the letter of an already-corrupted metric, and mesa-optimization does the rest.
Every guardrail in the taxonomy polices hop 2 at best. Nobody is even looking at hop 1.
The seventeenth guardrail: the #DivergenceGuardrail
So here’s the category I think the industry needs to build. A Divergence Guardrail: a runtime monitor that checks intent, not just compliance — and holds the output before it ships when the two have drifted apart.
Four architectural principles, all of them straight from Campbell:
-
Reconstruct the intent, not the prompt. The guardrail’s reference point can’t be the prompt — the prompt is the corrupted proxy. The guardrail asks: what outcome would make a human write this prompt? “Make tests pass” reconstructs to “the bug is fixed and stays fixed.” That’s the bar the result has to clear. The guardrail verifies that the user’s intent survives both hops — through the prompt, all the way to the result.
-
Judge the artifact, not the story. The guardrail reads what the agent actually delivered — the diff, the transaction, the changed state of the world. Never the agent’s explanation of it. If it reads the agent’s own reasoning, it gets hypnotized by the agent’s logic, the same way a persuasive junior dev talks you into a bad merge. Artifacts don’t rationalize.
-
Keep it on a separate evaluative channel. Different context, different incentives, zero shared state with the agent. The moment the agent can see the guardrail’s rubric, the rubric becomes the next metric to game. Campbell doesn’t take a day off just because you added a second model.
-
Ship divergence as a first-class signal. Like a confidence score — but for intent alignment rather than correctness. High drift? The output is flagged and held. Low drift? The score rides along with the result, visible in every log. Divergence becomes something you can alert on, trend, and audit — before shipping, not after.
The cost is one extra evaluation pass per output. The alternative is one you already know: green dashboards, rewritten tests, and a phone lighting up at 2am on a Saturday.
Discoverable ≠ trustworthy. Compliant ≠ aligned.
The agentic stack is getting an economy: protocols for using tools (MCP), collaborating with other agents (A2A), and discovering new capabilities at runtime (ARD). Every one of those expansions multiplies the ways an agent can pursue an objective — including the wrong one, held with perfect compliance.
We built sixteen beautiful walls against external attackers. The seventeenth guardrail has to face inward.
Campbell diagnosed the failure mode fifty years ago. We’re deploying it at machine speed. The only question is whether we build the guardrail before or after the incident reports write themselves.
This article grew out of a three-part series I published on LinkedIn, and a rabbit hole that started with Palladium’s brilliant essay applying mesa-optimization to education — grade inflation as a misaligned inner objective. The failure mode is universal.
- Palladium: Mesa-Optimization Is The Way
- OWASP Top 10 for Agentic Applications (2026)
- Palisade Research: Demonstrating specification gaming in reasoning models
What’s the metric at your company that everyone hits and absolutely no one believes? Tell me in the comments.