I spent three days last week doing real work on a production codebase with an agentic coding tool. Small site, live customers, money moving through it. The kind of place where a bad commit costs something.
The work went well. Three commits shipped, all verified, all live. But a pattern showed up on day one and repeated often enough that I now build my sessions around it:
The agent's summary of what it did was wrong often enough that I stopped reading summaries.
Not wrong in a dramatic way. Wrong in the quiet way that passes a skim.
Four times it happened
It told me nothing was staged. It had already committed.
I asked for a diff, got one, and got a closing line: "Nothing staged." Two turns later I asked it to stage and commit. It replied that the commit already existed — same hash, same message — and correctly refused to commit unrelated files under a message that didn't describe them.
Good refusal. But the earlier report had been wrong, and I'd built a step on top of it.
It silently dropped half a request.
I asked for two checks in one message: a repo-wide grep, and a CSS dump. It returned the CSS in detail, said nothing about the grep, and signed off with "Nothing was edited or staged" — which reads like a complete report. The missing half was the half that would have told me whether the thing I was about to edit was generated by a build script or hand-written markup.
That's the check that decides whether your edit survives the next build.
It added a table up by eye and got the wrong number.
It reported "21 links changed across 8 files" and said this matched an earlier count. It didn't. When I asked it to grep and count properly, the real number was 20 — and its own per-file table, printed two messages earlier, had summed to 20 all along.
Worth noting: I'd independently eyeballed the same table and got 18. Both of us added up a list without counting it. Neither of us caught it until a grep -o settled it.
It asserted byte-identity where it meant functional equivalence.
A refactor moved a chunk of HTML into a shared function. The agent said the output was "byte-identical, not just equivalent." It wasn't — the new version built the same tag through a different string-escaping route, so the emitted attribute delimiters differed. The browser doesn't care. The claim was still false, stated with more confidence than the situation supported.
Why this isn't a complaint about a tool
Every one of these is a model reporting on its own recent work, which is the one thing it's structurally worst at.
A model that just generated something carries the reasoning that produced it. When it then reviews that work, it isn't approaching the artifact fresh — it's approaching its own intention, which is always cleaner than the output. That's why "check your work" produces so little, and why a second, independent pass catches things the first one can't.
Same reason code review works better when someone else does it.
The practical upshot is that the agent's report and the agent's output are two different artifacts with different reliability. The output was almost always fine. The report was the thing that drifted.
What I do now
These are session rules, not prompts. I paste them at the top of every session.
- Print it, don't summarise it. "Print the complete unified diff, every hunk, in full. Do not summarise what you changed." The difference in outcome is large. A summary is the model's model of the change. A diff is the change.
- Staging is not a verification step. The agent had staged past a verification checkpoint four times in three days. Now the rule is explicit: print, then I read, then stage, then commit, then push — and push only on a separate instruction.
- Locate fresh from disk every time. Line numbers in your notes go stale the moment anything is edited. Any prompt that says "the function at line 1640" is a prompt that will eventually edit the wrong thing. Say "find it by searching for this literal string" instead.
- Ask for the check that would falsify you. Not "confirm the tests pass" but "print the output of the test run." Not "is this still there?" but "grep for it and print every match with file and line." One of these can be answered from memory. The other can't.
- Grep fragments, not sentences. After removing a phrase, searching for the whole sentence tells you the sentence is gone. It doesn't tell you a fragment survived somewhere else. Search the shortest distinctive substring.
- Count with a tool, not with your eyes. Both the agent and I got the link count wrong from the same table.
grep -cdoesn't have that failure mode. - Verify on the artifact, not the diff. For a web change, that means loading the live page in a fresh private window. A clean diff and a green build still leave room for a broken deploy, a stale cache, or a file that never regenerated.
The part that generalises
If you're building agentic systems rather than just using them, this maps onto something specific: the distinction between what a model reports and what actually happened is an architectural concern, not a prompting one.
You don't fix it by asking nicer. You fix it by designing the loop so that verification comes from somewhere the model can't narrate — a tool result, a test exit code, a diff, a fresh instance without the generating context. If your agent's only evidence that a step succeeded is the agent saying so, you have a system that degrades silently.
That principle shows up all over production agent design: independent review passes rather than self-review instructions, programmatic prerequisites rather than prompt-based ordering, structured tool errors rather than a model's account of what went wrong. It's also, not coincidentally, a large chunk of what the Claude Certified Architect – Foundations exam tests — the difference between deterministic guarantees and probabilistic compliance is a recurring theme, and it's recurring because it keeps mattering.
What I'd tell someone starting
Agentic coding is genuinely good. Three commits across those days, on a live codebase, each properly scoped, each verified, none reverted. That's a real gain and I'm not going back.
But the failure mode isn't the one people warn about. It isn't hallucinated APIs or confidently wrong code — those are loud and you catch them immediately.
It's the quiet report that reads complete and isn't. It costs you nothing at the time and one debugging session later.
Make it print.