Matt Pocock / diagnosing-bugs
Why do agent bug fixes so often fix the wrong thing?
Because a plausible fix is available long before the cause is understood.
What to remember
- Diagnosis and repair are different jobs. Mixing them is how symptom fixes ship.
- Handing over your theory early is the fastest way to get your theory confirmed rather than tested.
- Ask for the evidence that distinguishes the cause from the next most likely one.
- It overlaps with systematic-debugging from superpowers. Try both and keep one.
01What it changes in practice
Given a stack trace, an agent will almost always produce a change that makes the error disappear. That is not the same as fixing the defect, and the difference usually surfaces a week later with a stranger error.
Insisting on a stated cause before a change makes the reasoning inspectable. If the cause is wrong you can see it immediately, which is far cheaper than discovering it through a regression.
02Where it disappoints
It cannot diagnose what it cannot observe. Missing logs, no reproduction and unavailable environments limit it exactly as they limit a human.
On trivial bugs it is slower than just fixing them, so keep it for the ones that already wasted an afternoon.
Install and first run
- Install it with the skills CLI from mattpocock/skills, published under MIT.
- Give it the observation, not your theory. What you expected, what happened, and how to reproduce it.
- Require a stated cause and evidence before accepting any change.
Questions people actually ask
- How is this different from just asking for a fix?
- It requires a stated, evidenced cause before code changes, so symptom patches are visible instead of silent.
- Should I share my theory of the bug?
- Not at first. Give observations and a reproduction, or you will get your own theory confirmed back to you.
- How does it compare to systematic-debugging?
- They overlap. superpowers offers a four phase root cause process, this one is lighter. Running both is duplication.
Sources
Machine readable versions of this page