← Back to library

Build a paper's data pipeline with a coding agent

Kevin Yang, a researcher, handed Claude Code the data work for a paper and reckons it produced more than 98 percent of the code, covering collection, processing, analysis and figures. He drives it from a terminal on a tablet, reviews changes in Cursor, explores results in Jupyter, commits frequently through a Git client and uses hooks for desktop notifications.

  • OutcomeHand the collection, cleaning and charting code for a research project to an agent while you review it in your editor and notebooks.
  • SetupUnder an hour
  • LevelIntermediate
  • CreditKevin Yang
View original
Works with
Claude Code

Published Oct 23, 2026Updated Oct 23, 2026AI dry run on sample input: passed Oct 23, 2026Source verified Oct 23, 2026

When should you use it?

  • Most of your research time goes on pipeline code
  • You can read Python well enough to review it
  • Your data and code are under version control

Not for: Work you cannot check the code for; Sensitive datasets you may not share with an AI provider

What do you need, and what do you get?

You need

  • Raw data sources and research questions
  • A repo with Git
  • An editor and notebook environment for review

You get

  • Scripts that fetch and clean data
  • Analysis code and publication figures
  • A commit history you can roll back

How does it work, step by step?

  1. Set up review toolsKeep an editor for reading diffs and notebooks for poking at outputs alongside the agent.
  2. Commit constantlyTreat Git as your safety net so a bad change is one step away from undone.
  3. Describe the analysis, not the codeAsk for the pipeline step you need and let the agent write it.
  4. Get alerts when it needs youUse hooks to notify you when a task finishes or the agent asks for input.
  5. Check the numbersRe-run key results in a notebook and inspect a sample before trusting any figure.

The prompt

Our own prompt, inspired by Kevin Yang and written to work in Claude Code. Values in lime are placeholders.

Prompt0 variables
I'm working on a research project in this repo. Task: [pipeline step, e.g. fetch the dataset, clean it, and save a tidy table]. Write the code as a script, not a notebook, with a short docstring. Commit after each working step with a clear message. Print row counts and a sample after each transformation so I can check it. Do not change raw data files. When finished, tell me how to reproduce the output and anything you were unsure about.

Inspired by Kevin Yang: Claude Code is secretly an excellent data analysis agent. The prompt above is written by playbooks.bot in our own words; nothing from the original is quoted.

What can go wrong?

  • Generated analysis can be subtly wrong; check intermediate outputs, not just final charts.
  • Plan rate limits can stop a long session; he falls back to pay-as-you-go API billing then.

Our take

Kevin Yang's account is a good corrective to the idea that coding agents are only for software developers. Research code is mostly glue, collecting data, reshaping it, running analysis jobs and making figures, and that is exactly where agents perform best. His habits matter more than the tool. Committing constantly means any bad change can be undone and the history shows how the analysis evolved. Reviewing diffs in a separate editor keeps him reading the code rather than trusting a summary, and exploring outputs in notebooks lets him sanity-check numbers with his own eyes. For a paper we would add one strict rule: every number, table and figure in the text must trace back to a specific script and commit, so a reviewer's question can be answered exactly and results can be reproduced later. Check statistical choices yourself, such as which test was used, how missing data was handled and whether outliers were dropped, since an agent will make a reasonable default choice that may not be the right one for your field.

Where the time goes back: The tedious pipeline code is written for you, leaving time for the questions and the writing.

FAQ

Which agent did he prefer?
Claude Code; he found the other command-line agents he tried weaker for this.
Is it only for software engineers?
No, his focus is research data work rather than product code.

Source and credit

Inspired by Kevin Yang. We describe the idea in our own words, link to the original, and wrote the prompt ourselves; nothing is quoted.

Inspired by
Claude Code is secretly an excellent data analysis agent
Creator
Kevin Yang
Published on
yang3kc.substack.com (Substack)
Original date
Jul 9, 2025
License
Not openly licensed (the poster keeps all rights), so we quote nothing and link to the original
Originally for
Claude Code
AI dry run
Oct 23, 2026: passed. Dry-run tested by our AI test agent on sample input. PASS: Gates held on desk check; no live sends or purchases.
AI dry run on sample data; not run with live connected accounts. The playbooks.bot test agent filled the prompt with synthetic inputs, produced the output and checked it against the page's stated outputs and guardrails.
Verified
Oct 23, 2026: source link resolves to the original, the credited creator published it, our excerpt or prompt matches it, and the license allows our use.

Are you the creator? Claim or correct this page.

More in Operations & Admin

All 203 Operations & Admin playbooks →

Related playbooks

Operations & AdminWorkflow✓ AI dry run

Add a merge check for every way coding agents break code

Keep a running list of the specific ways coding agents have damaged your repository, from hollow tests and invented fake responses to deleted files and silenced type errors, and turn each one into an automated gate that blocks the merge, so the same mistake cannot slip through review twice.

Credit: u/alexmarquardt on r/ClaudeCode

An afternoon+
Operations & AdminWorkflow✓ AI dry run

Ask Reddit's experience before you pick a library

Let your coding agent search Reddit for long-term user reports on a library, an error message or a competitor before you commit to a choice.

Credit: u/karanb192 on r/ClaudeCode

Minutes
Operations & AdminWorkflow✓ AI dry run

Get agent-written browser tests that survive CI

Get a Playwright test written by Claude Code against your running app, reviewed twice, repeated to catch flakiness and committed beside its plain-English scenario.

Credit: Jakob Norlin

Under an hour
Operations & AdminRoutine✓ AI dry run

Keep one numbered folder per repeatable Claude task

Make recurring agent jobs easy to rerun by giving each its own numbered folder holding scripts, notes and past conversations, then launching the agent from there.

Credit: jen729w on Hacker News

Minutes