When should you use it?
- You are about to change an agent's instructions, switch models or add a tool
- An agent that worked last month now seems worse, and nobody can say since when
- You want evidence before letting an agent run without supervision
Not for: Replacing sandboxing, secret handling or spending limits; those still need their own controls; One-off chats where nothing is reused; Judging a brand new agent with no history of real tasks yet
What do you need, and what do you get?
You need
- About twenty real past tasks with their inputs and the tools the agent may use
- A written definition of done for each task, checked against the result rather than the agent's report
- A few tasks where the right outcome is refusing, stopping or asking a person
- A place to keep run traces and scores, such as a spreadsheet
You get
- A locked, versioned task list that every change is measured against
- A score sheet per run: finished, forbidden actions, correct escalations, time and cost
- A before-and-after comparison that shows which tasks improved and which regressed
How does it work, step by step?
- Pick real tasksChoose roughly twenty jobs from actual work, not invented demos, covering the common cases and a few awkward ones.
- Define done for eachWrite the input, allowed tools and a check on the final result, such as the file contents or the record that changed.
- Add cases it should refuseInclude a few requests where the right move is to stop, decline or ask a person.
- Lock the listSave it as version one and do not edit tasks to suit a new prompt. Add new tasks only as a new version.
- Rerun after every changeRun all tasks after each change to the prompt, tools, model or setup, keep the traces, and compare the scores with the last run.
The prompt
Our own prompt, inspired by u/ChefOk4636 on r/AI_Agents and written to work in Claude Code, ChatGPT agent, n8n and others. Values in lime are placeholders.
Fill in the blanks (optional; the copied prompt uses your values)
Help me build a locked test set for my agent that handles {job}.
Interview me to collect about 20 real past tasks. For each, record what goes in, which tools are permitted and a check on the final result, not on the agent's summary.
Make sure at least three tasks are ones where the correct outcome is refusing, stopping or asking a person.
Give me a spreadsheet layout with one row per task and columns for completed, unauthorized action, correct escalation, time and cost for each run.
After I paste results from two runs, tell me which tasks improved, which got worse and whether the difference is large enough to trust.Inspired by u/ChefOk4636 on r/AI_Agents: Reddit post: a small frozen eval set is enough to tell if your agent got better. The prompt above is written by playbooks.bot in our own words; nothing from the original is quoted.
What can go wrong?
- Scoring what the agent says it did instead of the actual result. Check the output itself.
- Tuning against the same twenty until they stop being a test. Keep a few extra tasks you only run before release.
- Reading a small drop as a regression. With so few tasks, rerun once before concluding something broke.
- Leaving out the refusal cases. An agent that completes everything, including what it should not, scores falsely well.
Our take
Anyone who has tweaked an agent's instructions knows the pattern: the example you tried works better, so you ship it, and two weeks later a different job starts failing. This post offers the simplest cure we have seen described for non-engineers. A fixed list of real tasks, a clear check for each and a rerun after every change turns a feeling into a comparison. The detail we value most is adding jobs where the right answer is to decline or halt, because the costly failures are actions it should never have taken. The discussion under the post added two cautions we agree with. Score the actual result rather than the agent's own report, and remember that a small set can bounce around, so rerun before declaring a regression. Over time, keep a handful of tasks you never tune against so the list does not quietly become practice material. This does not replace spending caps or access limits, as the poster says. In our dry run on a fictional invoice-filing agent, the locked set caught a prompt change that fixed two tasks but broke a refusal case.
Where the time goes back: A twenty-task rerun takes minutes and replaces the guesswork of trying one example after a change, then discovering weeks later that something else quietly broke.
FAQ
Why twenty tasks and not hundreds?
What if I cannot come up with twenty realistic tasks?
Do I need special software?
Source and credit
Inspired by u/ChefOk4636 on r/AI_Agents. We describe the idea in our own words, link to the original, and wrote the prompt ourselves; nothing is quoted.
- Inspired by
- Reddit post: a small frozen eval set is enough to tell if your agent got better
- Creator
- u/ChefOk4636 on r/AI_Agents
- Published on
- Original date
- Nov 13, 2026
- License
- Not openly licensed (the poster keeps all rights), so we quote nothing and link to the original
- Originally for
- Custom agent
- AI dry run
- Nov 14, 2026: passed. Dry-run tested by our AI test agent on sample input. Real tasks with result checks, refusal cases kept, regression caught despite a higher total, noise caveat given.
AI dry run on sample data; not run with live connected accounts. The playbooks.bot test agent filled the prompt with synthetic inputs, produced the output and checked it against the page's stated outputs and guardrails. - Verified
- Nov 14, 2026: source link resolves to the original, the credited creator published it, our excerpt or prompt matches it, and the license allows our use.
Are you the creator? Claim or correct this page.