“I tracked my time as an AI agent developer and discovered something deeply wrong.”
After logging my work for weeks, the numbers stared back at me: 10% on coding and design. 90% on one thing — manual testing.
Not automated testing. Not even manual testing in the QA sense. I’m talking about opening a browser, typing in a test dataset, watching the agent step through its workflow, and relying on gut instinct to decide whether the output was “better” or “worse” than before.
And that wasn’t even the worst part.
The worst part was that the AI agent itself is fundamentally non-deterministic. Change one line of a prompt — a single sentence — and the entire system could fall apart. And more often than not, I couldn’t even tell if it was broken without running it again. And again. And again.
At some point, I stopped being a developer. I became a human test harness for a system I could barely understand anymore.
Was this normal?
Could I turn that 90% into an automated, repeatable system that didn’t require me staring at a screen, clicking, and guessing?
The Verifier’s Rule — Why the 90% Problem Exists in the First Place
Before diving into solutions, I needed to confirm: was this just my problem?
It wasn’t. In 2025, Jason Wei — a researcher at OpenAI and Meta — observed a phenomenon he called the Asymmetry of Verification. Many tasks are hard to perform but easy to verify. A child can tell you whether a math solution is correct long before they can solve it themselves.
From this observation came what he called the Verifier’s Rule:
Any task that is solvable and easy to verify will eventually be solved by AI.
What does this have to do with my 90%?
I was spending 90% of my time on verification precisely because, for the AI agents I was building, generation had become cheap but verification remained expensive. That asymmetry — cheap generation, expensive verification — was the entire problem.
The implication was clear. If I could transform “is this agent correct?” from a gut-feel, eyes-on-screen judgment into an automated, repeatable verification system, I would be converting a hard-to-verify problem into an easy-to-verify one. And once it became easy to verify, the Verifier’s Rule dictated that AI itself could take over the rest.
My role would shift from “person who writes code” to “person who designs verifiable architectures.”
An Industry That Solved This 30 Years Ago
This sounds abstract. But here’s the thing — another industry cracked this problem three decades ago.
That industry is semiconductor design and verification. Which happens to be the industry I work in.
Here’s a fact that surprises most software engineers: there are more engineers verifying chip designs than there are engineers writing them. Before a chip tapes out, the verification effort routinely consumes 2× to 3× the resources of the design effort. Sometimes more.
Why? Because once a chip is manufactured, there’s no “hotfix.” No git push. One bug costs millions of dollars. Gone. Irreversible.
The entire industry was forced by economic reality to pour its energy into one thing: finding every possible bug before it becomes silicon. And the core competency of that work isn’t “being good at writing Verilog.”
It’s:
- Designing regression test suites that expose bugs proactively
- Knowing where to embed assertion checkpoints
- Measuring coverage — and knowing when you’ve measured enough
Sound familiar?
When AI makes writing code dramatically cheaper — the way EDA tools made writing Verilog easier — your value stops being about the code you write. It becomes about the verification architecture you can design around it.
Building the Verification Architecture — Step by Step
So how did I actually apply this to our AI agent?
Step 0: Make Your Agent “Agent-Friendly”
The first thing I did was not what you’d expect. I didn’t write a single test.
Instead, I rewired our AI agent so that another AI agent could interact with it.
I had two systems on my desk:
- Our AI agent — the one I needed to test
- My coding tool (Codex, etc.) — the one I used to build
The reason testing had been so painful was obvious in retrospect. Our agent’s only interface was a web UI — designed for human eyes and human hands. To test it, a person had to sit there, click, type, and watch. It was built for humans from day one.
“AI agent-oriented development” — a phrase we hear constantly — actually boils down to three concrete questions:
- Can your system run when disconnected from human eyes and a UI?
- Does it leave enough intermediate state for another AI agent to understand what happened?
- Have you designed dedicated interfaces that let AI autonomously retrieve that state?
Concrete example: when our agent runs a workflow, I need Codex to see the full execution trace — which tools were called, what data came back at each step, where the branching happened. If that information evaporates the moment it leaves the UI, you’ve built an interface for humans, not for AI.
MCP or Decoupling?
There are two approaches here.
Path A: MCP-based UI emulation. Use the Model Context Protocol to let an AI agent drive the browser, observe the DOM multimodally, and simulate human interaction.
Path B: Full frontend-backend separation. Decouple the backend into a standalone AI agent that runs independently, with all operations available as CLI commands. The frontend becomes a thin monitoring layer — it visualizes backend state for humans but isn’t required for execution.
I chose Path B. The reason is straightforward: for authentication flows, corner cases, and execution efficiency, direct decoupling is significantly faster and more robust than screen-driving through MCP. You can see the DOM, yes — but you’re still fighting the fact that the system was fundamentally designed for human interaction.
To be honest, this decoupling took real effort. We had built heavily on Vercel’s AI SDK, which was fantastic for rapid initial development but required non-trivial restructuring to cleanly separate frontend and backend. The result, though, was worth it: a backend that can run independently, stably, for extended periods — no browser required.
ACI: AI Agents Are a New Category of User
We’re not alone in this conclusion. Princeton’s SWE-Agent paper introduced the concept of ACI — Agent Computer Interface. Their core argument:
AI agents are a fundamentally new kind of user, distinct from humans. They require purpose-built interfaces.
And critically, their experiments showed that interface design quality had a larger impact on agent performance than switching to a more powerful model. Read that again.
Every time you hear “AI agent-oriented programming” or “agent-native services,” the essence isn’t a particular technology. It’s this: can another AI — or more concretely, can Codex — rapidly and autonomously access the information it needs from your system?
Step 1: Build a Two-Layer Judge
Once the system is agent-accessible, we can actually start building tests. At its core, this is regression testing. The concept isn’t complex. What’s complex is applying it to a non-deterministic system.
Let’s use a concrete scenario: adding a new tool to our AI agent. As the developer, I have a clear expectation — “under these conditions, this tool should trigger, and afterwards the workflow should follow this path.” This expected path is the Happy Path, and it’s the acceptance criterion for the new feature.
The immediate problem: who judges?
An agent’s output is a block of natural language. An execution trace. It’s not a numeric value you can diff. Binary pass/fail classification doesn’t cut it.
My approach: split the judge into two layers.
Layer 1: Deterministic Assertions
These are hard checks — binary, mechanical, zero-cost:
- Did the expected tool actually get invoked during this test?
- At step N, how many records exist in the intermediate state?
- Which branching conditions were covered?
This layer is 100% reliable, fully deterministic, and costs nothing to run. No LLM calls needed.
Layer 2: LLM Supervisor
For the fuzzy parts that assertions can’t capture, we bring in an LLM judge — I call it the Supervisor.
Critical design decisions here:
The Supervisor’s context must be pristine. It does not participate in development. It has zero knowledge of the codebase, the design rationale, or the implementation history. Its eyes see only one thing: the specification of what “correct” looks like.
For outputs without a single correct answer, don’t ask for right/wrong. Ask for a score. The Supervisor rates outputs quantitatively — “78/100 on relevance, 62/100 on conciseness.” Scores make it possible to track direction: did this change make things better or worse, and by how much?
If you’ve worked in chip verification, this architecture will feel familiar. Assertions. Coverage metrics. Scoreboards. These are standard hardware verification primitives, adapted for a system that’s inherently more probabilistic than silicon. The key difference: an AI agent requires a higher tolerance for uncertainty, so the thresholds and score interpretations need to reflect that.
Step 2: Accumulate Test Cases Relentlessly
With judges in place, the next phase is case accumulation.
Every new feature adds at least one Happy Path test. As the feature set grows, Happy Path coverage grows organically with it.
But Happy Paths alone aren’t enough. You also need to harvest inputs you never designed for — the chaotic, diverse, real-world inputs that come from actual users and exploratory testing. Feed them into your test suite. These are the cases that surface the assumptions you didn’t know you were making.
Together, these two categories of regression tests ensure:
- The system doesn’t regress as it evolves
- Blind spots are systematically minimized
Step 3: Feature Flags as a Comparison Lens
One practice I consider non-negotiable: every new feature ships with a kill switch.
A feature flag gives you two versions — ON and OFF — that can be run through the exact same regression suite. The result is clarity that intuition can never provide:
- What did the new feature actually improve? (Which scores went up?)
- What did it silently break? (Which scores went down?)
Without this, you’re stuck in the land of “it feels better now,” which is not engineering — it’s wishful thinking.
Closing the Loop — From “Writing Code” to “Defining Correctness”
Once this architecture is in place, the most elegant step becomes possible: closing the loop.
Instruct your coding agent (Codex, or equivalent) to follow this cycle:
- Design the regression tests first — define expected behavior
- Design the feature flag — set up the comparison baseline
- Implement the code
- Run regression tests with flag OFF, then ON
- Feed test results back into code revision
- Repeat until tests pass
There is no human in this loop. The coding agent, operating within the verification architecture you’ve built, self-corrects until it converges on a solution that satisfies the defined correctness criteria.
At this point, we are no longer writing code. We are no longer manually testing. We are defining what “correct” means. The loop handles everything else.
This is the Verifier’s Rule in action: make it verifiable, and AI takes over the rest.
The 90% — the manual verification that once consumed my working life — becomes genuinely automated.
Don’t Copy This. Find Your Own 90%.
Everything I’ve described — frontend/backend decoupling, two-layer evaluation, regression test accumulation, feature flags — is, at the end of the day, just one example. One team’s answer to one specific bottleneck.
I’m not here to tell you to do exactly what I did. Frontend/backend separation and regression testing are well-documented practices. Every technical blog has covered them. What I want you to see is why we made these decisions — the reasoning that led from “this is painful” to “this is the architecture that fixes it.”
If that 90% inefficiency hadn’t backed me into a corner, I probably wouldn’t have made these technical choices. It was the pain that forced the structural answer.
So here’s what I want you to take away:
- Is there a similar bottleneck in your own engineering work?
- Is something quietly eating your time — something repetitive, something that makes you frustrated every time you touch it?
- If there is, are you treating it as inevitable, or are you treating it as a design problem?
If you find it, solve it your way. My approach is just a starting point for thinking. It might be worthless to you — or it might spark an idea that fits your context perfectly. Either outcome is fine.
”Should I Use AI to Write Code?” Is No Longer the Question
A year ago, developers debated: should I use AI to write code?
In 2026, nobody’s debating this anymore. Everyone is using AI.
The question has shifted:
“How do you use AI in a way that makes you more efficient than everyone else using AI?”
If a task that took ten minutes now takes two hours with AI — with the same result — there is no efficiency gain. AI made you slower. That’s not progress. That’s cargo-culting.
The difference between AI as a force multiplier and AI as a productivity trap isn’t the model you’re using. It’s the architecture you’ve built around it.
I found my bottleneck. I built a system around it. I got my 90% back.
Now go find yours.
NeoAnalogLab provides end-to-end consulting on AI agent development efficiency, verification architecture design, and autonomous development workflows. If you’re hitting the “manual testing wall” with your AI agents, we’d be happy to talk.