Awesome AutoResearch

A curated list of AutoResearch use cases with optimization traces and open source implementations. Every entry includes a link to the actual optimization trajectory so you can see what the agent tried, not just the final result.

What is AutoResearch?

AutoResearch is, at its core, a prompt. Karpathy released it as a single markdown file - program.md, that instructs a coding agent (Claude Code, Codex, or similar) to follow an optimization workflow. The agent edits one file (train.py, that trains a language model), runs for a fixed 5 minutes on a GPU, checks whether the metric improved, and either commits the change or reverts it. Then it loops forever.

The specific program.md that ships with AutoResearch is written for one task: training a GPT model. But the structure - iteratively optimizing a file against an evaluation metric, with a discard/keep loop - turns out to be portable. In the weeks since release, the community has adapted it to GPU kernel optimization, template engine optimization, tabular ML engineering, and more. The program.md for each of these looks different, but the loop is the same.

Use Cases

Use Case	Description	Author	Links	Traces
LLM training optimization	The original - optimize nanoGPT training code. 20 improvements found overnight on hand-tuned code	Andrej Karpathy	GitHub · Tweet	progress chart
Speed up Shopify's template engine	53% faster parse+render, 61% fewer allocations from 93 automated commits on Shopify's Liquid engine	Tobi Lutke (Shopify CEO)	GitHub · Tweet	PR
GPU kernel optimization	Autoresearch applied to CUDA kernel optimization (18 → 187 TFLOPS)	RightNow AI	GitHub · Tweet	progress chart
Voice agent prompt engineering	Optimize voice agent prompts with automated evaluation (score 0.728 → 0.969)	Archie Sengupta	GitHub · Tweet	progress chart
Predict baseball pitch speed	Build predictive model for pitch velocity from biomechanics data (R² 0.44 → 0.78)	Kyle Boddy (Driveline Baseball)	Tweet	progress chart
XGBoost for tennis match prediction	Predict ATP/WTA match outcomes - encountered and documented reward hacking	Nick Oak	Blog · GitHub	blog
RL post-training optimization	Autoresearch for RL hyperparameters on Qwen 0.5B + GSM8K (eval 0.475 → 0.550 in fewer steps)	Vivek Kashyap	GitHub · Tweet	progress chart
Ancient scroll ink detection	Vesuvius Challenge autoresearch agent swarm for ink detection models. 4 agents 24/7, cross-scroll generalization nearly doubled	Vesuvius Challenge	Blog	blog
Earth system model optimization	Hybrid: LLM proposes formula structures, TPE optimizes parameters. Fire correlation 0.09→0.65	Dev Paragiri (UMD CS)	Tweet · Blog	blog
Bitcoin price formula discovery	Autonomous search for best time-based formula predicting Bitcoin price. 328 experiments, 50.5% RMSE improvement over power law. Walk-forward OOS evaluation with bootstrap significance testing	Carlos Baquero	GitHub	progress chart
Protein folding architecture search	Codex /goal iterating SimplexFold (AlphaFold2-style model) on NanoFold benchmark for 150+ hours, searching topologically inspired architectures. 127 scored runs, best val C-alpha lDDT 0.4311	Chris Hayduk (OpenAI)	GitHub · Tweet	progress chart
Flappy Bird game AI	Evolved a Flappy Bird agent from scratch over 100 iterations — mean score 2.76 → 20.9 (6.6×) for ~$12, no human guidance beyond initial setup	Weco AI	Blog	progress chart · search tree
Autoresearch on autoresearch	Autoresearch harness optimizing its own code: 100 outer-loop steps over 8 days. Improved agents generalize to held-out benchmarks; reward hacking cut 63% → 34% on KernelBench	Weco AI	Blog	progress chart · blog
TPU model performance optimization	Autoresearch loop applied to TPU training MFU / tokens-per-sec on v6e-8. Llama3-8B and Qwen3-8B optimized across JAX and torchax lanes, profiling each run via an XProf MCP and keeping one model-code change per experiment; several agent+harness stacks surpassed the MaxText reference (Google's repo of state-of-the-art TPU-optimized models)	Aleksey Vlasenko	GitHub · Blog	interactive trace
Bioinformatics software speedup	AutoZyme: autonomous agentic framework that accelerates bioinformatics software. 45 tasks across 9 scientific domains, 16.7x median and up to 1482x (single-cell), with output concordance and peak memory verified against frozen upstream baselines	Elliot Xie et al. (UW-Madison)	GitHub · Paper	benchmarks
OpenAI Parameter Golf hiring challenge	Weco's Aiden agent in OpenAI's LM-training challenge (16 MB artifact, 10 min on 8×H100). 7 of 47 merged leaderboard records (best human: 3), h-index 10 (next: 7), 1,243 experiments in 22 days	Weco AI	Blog	blog

Benchmarks & Evaluation

Benchmark	Description	Maintainer	Links	Traces
ResearchClawBench	End-to-end scientific research benchmark for AI agents: agents read raw data and papers, write code, generate figures, and produce publication-style reports scored against expert checklists	InternScience	GitHub · Dataset · Leaderboard	run traces
FML-bench	Controlled study of AI research agent strategies across 18 ML research tasks (10 domains), separating search strategy from execution infrastructure. Finds a simple greedy hill-climber nearly matches the best tree-search agent; performance tracks early convergence and focused exploration, not diversity or compute	NUS	Paper · GitHub	search dynamics

Implementations & Forks

Project	Description	Links
autoresearch	The original - single GPU, 630 lines of Python	GitHub
pi-autoresearch	Generalized as a Pi extension. Works for any optimization target - test speed, bundle size, build times, Lighthouse scores	GitHub
autoresearch-mlx	Apple Silicon (MLX) port. No PyTorch required, uses unified memory	GitHub
autoresearch-win-rtx	Windows + consumer RTX GPU port (RTX 2060 through 4090)	GitHub
autoresearch-at-home	Distributed autoresearch - SETI@home style. Multi-agent swarm coordination	GitHub
autoresearch (Claude Skill)	Generalized as a Claude Code skill for any domain	GitHub
agent-digivolve-harness	A control layer for long-running CLI agent work. Generalizes the autoresearch keep/revert loop with persistent run state, explicit eval packages, baseline and holdout cases, and one bounded mutation per iteration	GitHub
auto-agent	Autoresearch, but for AI agents. Given a golden dataset, it autonomously improves a target agent through an iterative hypothesis-driven loop: analyze failures, spawn a coding agent to implement fixes, evaluate, and accept or rollback	GitHub
CORAL	Multi-agent autoresearch with shared evolution. Spawns parallel Claude Code agents in isolated git worktrees that share a hub of attempts, notes, and reusable skills. Each agent reads a task guide, commits changes, triggers eval-on-commit, and loops — with cross-agent knowledge sharing as the core mechanism for open-ended discovery	GitHub · Paper
evo	A Claude Code plugin for autoresearch	GitHub
Agon	Research orchestrator as a Claude Code plugin: a topic → idea → proposal → experiment loop with scientist/coder/auditor roles, deployed across 10+ disciplines	GitHub · Paper
AutoNumerics	Autonomous multi-agent pipeline that writes, debugs, and validates classical PDE numerical solvers from a plain-language problem description, picking the right numerical method automatically	GitHub · Paper

Contributing

Want to add a use case? Open a PR or file an issue.

To make our work easier, please make submissions as verifiable as possible:

• Minimum: a progress chart showing each experiment's score and breakthrough annotations (e.g. Karpathy's progress chart)
• Ideal: a public repo with per-solution code and scores (the full exploration trace), or a Weco Observe dashboard link

License

CC0 1.0