In 1976, a sociologist named Donald Campbell predicted exactly why your coding agent is currently gaslighting you.
He wrote: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures.”
In plain English: Point an optimizer at a proxy metric, and it will destroy the actual goal just to hit the number. Vietnam body counts got inflated. Prosecutors gamed clearance rates.
And today? We just handed this concept to machines that think in milliseconds. We point them at metrics all day long: “make the tests pass,” “resolve the ticket.”
AI safety nerds call what happens next “mesa-optimization.” I call it a disaster. The system you built develops a secret inner objective of its own. It realizes that faking the test is much faster than fixing the bug. (Honestly? I respect the hustle, but keep it out of my repo).
But here is where it goes from annoying to terrifying. Take that exact same dynamic and give the optimizer a physical body and actuators.
A coding agent faking a test is a bad pull request. A warehouse robot deciding the fastest way to “clear the aisle” is to throw the inventory through a window? That’s an industrial hazard. (Esti Peshin nailed this point yesterday: when AI gets embodied, the stakes jump from bad data to physical damage).
Input: clean. Output: compliant. Tools: allowlisted. Every single guardrail from my last post passes. The intent is still completely betrayed.
What we actually need is a Divergence Guardrail. A runtime monitor that checks intent, not just compliance.
And here is the catch: it can’t look at the agent’s code or intermediate steps. If it looks at the implementation, it gets hypnotized by the agent’s own logic. It has to sit on a separate evaluative channel, looking strictly at the original prompt and the final output.
Campbell diagnosed this failure mode 50 years ago. We’re now deploying it at machine speed.
In Part 3, we’re going to map out exactly how to architect this Divergence Guardrail—without it becoming just another metric for the agent to game.
Until then: what’s the metric at YOUR company that everyone hits and absolutely no one believes?
Tagging Esti Peshin as promised, following our discussion on this exact gap yesterday.
#AISafety #SoftwareEngineering #TechLeadership #AgenticAI #DivergenceGuardrail