Andrej Karpathy just open-sourced a project that lets humans sleep through the ML experimentation loop entirely. His new open-source autoresearch project already has more than 7,000 forks thanks to the simplicity of its concept and execution: give an AI agent a real (but small) language model training setup, a single file it's allowed to edit, a fixed five-minute time budget per experiment, and one metric to optimize.
Let it run. Come back to results. Repeat.
How Autoresearch works: The agent loop
The repo is built around three files with the agent touching only one.
prepare.py— Downloads training data, trains a BPE tokenizer, and provides the dataloader and eval utilities. The agent never touches this.train.py— The file the agent edits. It contains the full GPT model definition, optimizer (Muon + AdamW), and training loop. Architecture, hyperparameters, batch size, and attention patterns are all in scope.program.md— Your interface to the system via a Markdown file with research direction and instructions. You iterate on this over time, effectively building a research organization in a text file.
The agent reads program.md, proposes a change to train.py, runs a five-minute training experiment, checks val_bpb (validation bits per byte are vocabulary-size-independent, so architectural changes stay comparable), and either commits or discards the result. Then it loops.
Five minutes per experiment. ~12 experiments per hour. ~100 experiments overnight.
That volume of iteration would take a human researcher days or weeks.
Fixed time budget matters
Excluding startup and compilation, every experiment runs for exactly five minutes to achieve two things:
- Results directly comparable regardless of what changed on your hardware. A larger model for five minutes competes on equal footing with a smaller model for five minutes.
- Optimization for the best model achievable within your specific compute budget, not in the abstract.
The progress.png in the repo shows the val_bpb curve across agent-driven iterations. The improvements are real and measurable. The agent makes net-positive modifications to training code without supervision.
What this means for ML researchers and developers
High-throughput experimentation historically requires large teams, large compute budgets, or both. Autoresearch changes the math to be one GPU, one overnight run for 100 experiments.
The open source community has already extended the project to MacBooks, Windows machines with RTX cards, and AMD GPUs. Karpathy has published guidance for CPU-only setups.
Run autoresearch on Lightning Studios
Don’t have an NVIDIA H100 sitting around? The fastest path is to run autoresearch on Lightning Studios.
Lightning Studios gives you on-demand access to our owned fleet of NVIDIA GPUs (through Voltage Park) so you don’t have to manage infrastructure. You spin up a studio, clone the repo, and run the four setup commands:
curl -LsSf https://astral.sh/uv/install.sh | sh # install uv (skip if installed)
uv sync # install dependencies
uv run prepare.py # download data, train tokenizer (~2 min)
uv run train.py # verify setup with one experiment (~5 min)Once that's verified, open a coding agent in the repo directory and prompt it: "Have a look at program.md and let's kick off a new experiment."
Spin up a studio, kick off the loop, close your laptop. Check results in the morning.
For enterprise teams
If you're running an ML research function, autoresearch forces teams to think about optimizing their resources and attention. The bottleneck in most ML research cycles isn't compute. It’s the human review loop between experiments. Autoresearch doesn't replace researchers. It eliminates the waiting.
Build with Lightning Studios adds collaboration, audit logs, persistent storage, and team access controls on top of your on-demand GPU infrastructure.
Talk to us about enterprise GPU access →
Read more blogs and case studies
• ComfyUI in the Cloud for Agencies and Creative Teams
• Building Reliable Agent Systems with NVIDIA Nemotron 3 Super on Lightning
• View all