Our team built a dedicated bug-fixing agent called Sherlock.
It takes bugs directly from Slack. Over a recent 15-day window, it completed 13% of all company tasks. Of its last 100 assigned tasks, 90 finished successfully.
We did not get there by writing a more elaborate system prompt. We built a controlled workflow around the agent.
1. Ten fixed stages instead of a role prompt
A prompt such as "You are an expert engineer" can produce a different process every time. Sherlock follows a 10-stage state machine and cannot skip stages. Its diagnosis stage cannot touch product code.
2. Two explicit approval tokens
The first workflow had 7 approval checkpoints. That created a bottleneck for the engineer, so we reduced it to:
3. Artifacts, not chat claims
Each stage must produce concrete files: diagnosis.md, idea.md, plan.md, and review-report.md.
A message saying "diagnosis complete" does not count as completion.
4. State recovery from state.json
The workflow state is persisted to disk. If a session is interrupted, the next run reloads it. We tried reconstructing state from conversation history, but the agent hallucinated progress.
5. A six-iteration review limit
Sherlock reviews its own diff as an independent reviewer. It patches blocking issues and reviews again, but the loop stops after 6 iterations and escalates to a human.
6. Screenshots where decisions happen
For UI changes, before/after screenshots appear directly in the conversation rather than as file paths.
Example: "list not scrollable"
Slack → structured task → 14-minute autonomous diagnosis → 3 decisions for a human → approval → 37-minute implementation → PR + preview URL + screenshots.
Two human decisions total.
The 90/100 result applies only to tasks routed to Sherlock. It is not a universal bug-fixing success rate, and reopen-rate data has not been published.
Full write-up:
https://dev.to/yukioikeda/how-to-build-an-ai-agent-that-fixes-bugs-reliably-practice-in-sharkly-o63
What has worked best for your coding agents: explicit state machines, approval tokens, persistent state, or bounded self-review?
It takes bugs directly from Slack. Over a recent 15-day window, it completed 13% of all company tasks. Of its last 100 assigned tasks, 90 finished successfully.
We did not get there by writing a more elaborate system prompt. We built a controlled workflow around the agent.
1. Ten fixed stages instead of a role prompt
A prompt such as "You are an expert engineer" can produce a different process every time. Sherlock follows a 10-stage state machine and cannot skip stages. Its diagnosis stage cannot touch product code.
2. Two explicit approval tokens
The first workflow had 7 approval checkpoints. That created a bottleneck for the engineer, so we reduced it to:
- [START DEVELOPMENT] — allows code changes.
- [SUBMIT FOR TESTING] — allows push/merge.
3. Artifacts, not chat claims
Each stage must produce concrete files: diagnosis.md, idea.md, plan.md, and review-report.md.
A message saying "diagnosis complete" does not count as completion.
4. State recovery from state.json
The workflow state is persisted to disk. If a session is interrupted, the next run reloads it. We tried reconstructing state from conversation history, but the agent hallucinated progress.
5. A six-iteration review limit
Sherlock reviews its own diff as an independent reviewer. It patches blocking issues and reviews again, but the loop stops after 6 iterations and escalates to a human.
6. Screenshots where decisions happen
For UI changes, before/after screenshots appear directly in the conversation rather than as file paths.
Example: "list not scrollable"
Slack → structured task → 14-minute autonomous diagnosis → 3 decisions for a human → approval → 37-minute implementation → PR + preview URL + screenshots.
Two human decisions total.
The 90/100 result applies only to tasks routed to Sherlock. It is not a universal bug-fixing success rate, and reopen-rate data has not been published.
Full write-up:
https://dev.to/yukioikeda/how-to-build-an-ai-agent-that-fixes-bugs-reliably-practice-in-sharkly-o63
What has worked best for your coding agents: explicit state machines, approval tokens, persistent state, or bounded self-review?