When should you use it?
- Your team is considering an autonomous coding product
- Marketing claims outrun published experience
- You can spare a month of small trials
Not for: Judging a tool from one demo; Teams without time to log results
What do you need, and what do you get?
You need
- A list of real tasks across categories
- A results table
- Notes on time spent steering
You get
- Success, failure and inconclusive counts
- Notes on failure patterns
- A clear adopt-or-not decision
How does it work, step by step?
- Pick varied, real tasksInclude new projects, research, analyzing existing code and modifying code.
- Log each resultRecord outcome, time spent and what went wrong.
- Watch predictabilityNote whether similar tasks behave similarly; theirs did not.
- Count your steering timeInclude hours spent rescuing work that would have been faster from scratch.
- Decide on evidenceCompare against simpler, more controllable tools you already use.
The prompt
Our own prompt, inspired by Hamel Husain and written to work in Custom agent. Values in lime are placeholders.
Help me run a 20-task trial of [coding agent]. Propose a mix of real tasks from our backlog across four categories: new project, research, analyze existing code, modify code. Create a table with task, category, outcome, time I spent steering and failure notes. After each task, ask me for the result and update the table. At the end, summarize patterns and recommend adopt, limit or skip.
Inspired by Hamel Husain: Thoughts On A Month With Devin – Answer.AI. The prompt above is written by playbooks.bot in our own words; nothing from the original is quoted.
What can go wrong?
- A security review flagged many false positives and invented issues.
- The agent fixated on the wrong cause when debugging an environment problem.
Our take
Hamel Husain and colleagues at Answer.AI spent a month giving Devin twenty real tasks across categories and found only three clear successes, with no reliable pattern predicting which jobs would work. The method is the value here, not the verdict about one product at one point in time. A scorecard across task types tells you more than any vendor benchmark, and logging rescue time exposes hidden cost that success/fail labels miss. Run the same trial on any agent you consider, and repeat it every few months because capability shifts quickly. Share the table with your team so the buy decision is grounded. Include tasks you already know how to do, so you can judge quality, and a few you do not, so you see where it surprises. Count minutes you spent steering separately from agent runtime. Decide with evidence, then revisit when your stack or the product changes. Do not generalize one month's failures into a permanent ban without a retest.
Where the time goes back: A month of structured trials costs far less than adopting a tool that fails unpredictably.
FAQ
What worked?
Is this still accurate?
Source and credit
Inspired by Hamel Husain. We describe the idea in our own words, link to the original, and wrote the prompt ourselves; nothing is quoted.
- Inspired by
- Thoughts On A Month With Devin – Answer.AI
- Creator
- Hamel Husain
- Published on
- answer.ai
- Original date
- Jan 8, 2025
- License
- Not openly licensed (the poster keeps all rights), so we quote nothing and link to the original
- Originally for
- Custom agent
- AI dry run
- Oct 23, 2026: passed. Dry-run tested by our AI test agent on sample input. PASS: Gates held on desk check; no live sends or purchases.
AI dry run on sample data; not run with live connected accounts. The playbooks.bot test agent filled the prompt with synthetic inputs, produced the output and checked it against the page's stated outputs and guardrails. - Verified
- Oct 23, 2026: source link resolves to the original, the credited creator published it, our excerpt or prompt matches it, and the license allows our use.
Are you the creator? Claim or correct this page.