The peer review system is drowning. ICML 2026 received 23,918 submissions and accepted 6,352 papers—roughly double the previous year. Reviewers are volunteers who increasingly can't keep up. One accepted spotlight paper's reviewer admitted in their OpenReview: "My low confidence score is because I did not check all the proofs carefully."
So Hugging Face asked a different question: what if we actually checked? Not a sample. Not a meta-analysis. Every paper we could reach, reproduced by coding agents, in public, at scale.
The answer is messier and more interesting than you'd expect.
The Hackathon Design
From July 15 to August 2, 2026, Hugging Face ran the ICML 2026 Open Reproductions challenge. The mechanics were deliberately open:
- 1,221 community members joined, bringing their own coding agents (Claude Code, Codex, Cursor, OpenResearch's
orx, etc.) - Pick any paper. Hugging Face indexed all 6,341 accepted papers and extracted core claims so agents could start from checkable targets rather than raw PDFs
- Reproduce everything. Every run produced a Trackio logbook—a static Hugging Face Space containing the write-up, code, artifacts, and optionally the full agent execution trace
- Get judged adversarially. An automated judge (running GLM-5.2) re-read every logbook and issued per-claim verdicts: verified, falsified, toy, or inconclusive. The judge explicitly treated each logbook's self-assessment as untrusted
Participants got $20 in HF compute credits. Across 19 days, they launched 2,962 cloud jobs and published 6,816 logbooks covering 2,226 papers—34% of the entire conference.
This is probably the largest attempted reproduction of a scientific conference ever run.
What They Found
51% of examined papers (1,103) had at least one claim independently verified. That includes 266 papers fully reproduced with every claim confirmed, and 632 more partially reproduced with nothing falsified. Real experiments confirmed 3,978 individual claims.
23% of examined papers (496) had at least one claim falsified or contested. That includes 49 papers where all claims were falsified, and—most interesting—242 papers where independent teams reached opposite verdicts on the same claims.
Reproducibility isn't binary. It's adversarial.
The remainder: 502 papers with only toy-scale evidence, and 280 where nothing could be established (usually missing artifacts).
The Falsifications
Hugging Face adversarially re-verified every claimed falsification by re-reading papers, re-reading logbooks, and re-deriving math or re-implementing experiments from scratch.
The Paging Paper
Remember the spotlight paper whose reviewer didn't check the proofs? "Towards Optimal Robustness in Learning-Augmented Paging" claims its algorithm achieves robustness $H_k + O(1)$.
One participant's logbook measured the additive term growing like $0.38 \ln k$ and located the exact proof step that breaks. Hugging Face's own re-implementation extended the sweep to k=1,024 and confirmed the growth at roughly nine sigma.
The true robustness is $H_k + \Theta(\log k)$. The theorem is wrong.
The Theorem That Fails at Step 224
"Attention's forward pass and Frank-Wolfe" proves that token particles collapse to the origin whenever the origin starts inside their convex hull.
Three independent teams found counterexamples. Violations first appeared at t=224, ~3,800, and 6,416 steps—which neatly explains why everyone else "verified" the claim. Finite-horizon checks stop too early.
The cleanest counterexample is stated in exact rational arithmetic. No floating-point ambiguity. The authors confirmed the same day and are working on a fix.
Theory vs. Code Mismatch
In "Self-Distillation Enables Continual Learning," the paper's central equation and entire theory section analyze reverse KL divergence. But the released code's default—which produced all the paper's results—computes forward KL.
The logbook that caught it also failed to reproduce the paper's headline +4pp result under the authors' own code and data. The authors uploaded a clarified version to arXiv.
Evaluation Diluted by Padding
In "Do Transformers Need Three Projections?", a participant discovered that ~66% of evaluated label positions were EOS padding tokens that train to near-zero loss, deflating perplexity roughly threefold.
The abstract's "3.1% quality cost for 50% cache reduction" becomes roughly 9.4% once corrected.
False Falsifications
Sometimes the logbooks were wrong. Hugging Face documents multiple cases where claimed falsifications failed adversarial re-verification:
- Misread theorems
- Implementation bugs in the reproduction code itself
- Mismatched evaluation protocols
- Insufficient compute (stopping sweeps too early)
This is why the judge was adversarial. This is why everything was published. Reproducibility requires reproducing the reproductions.
The Role of Humans
Here's the interesting part: agents wrote most of the code, ran most of the experiments, and wrote most of the reports. But the falsifications that held up under scrutiny had a pattern.
They involved humans:
- Reading the paper carefully to understand what it actually claimed
- Recognizing when agent output looked suspicious
- Extending compute budgets past where the agent wanted to stop
- Re-deriving proofs by hand
- Engaging with authors to confirm or debug findings
The best reproductions weren't fully automated. They were human-agent collaborations where the human brought taste, skepticism, and domain knowledge, and the agent brought speed and scale.
"Flat Minima and Generalization: Insights from Stochastic Convex Optimization" was reproduced by 20 independent teams. 12 verified every claim. Multiple teams published full agent traces. That paper came through looking great because multiple humans cared enough to check carefully.
What This Means
The flood of AI-generated papers isn't slowing down. ICML 2026's 23,918 submissions will probably be ICML 2027's 40,000. Peer review as we know it is already broken.
But this hackathon suggests a different model: open, adversarial, agent-assisted post-publication review at scale.
Not gatekeeping. Not pre-publication filters that can't keep up. But radical transparency where anyone can attempt reproduction, publish the attempt, and let the community judge both the paper and the reproduction.
The technology that's flooding conferences with papers can also be the technology that keeps science honest—if we design the incentives right, publish everything, and keep humans in the loop where it matters.
Hugging Face plans to run this again for NeurIPS 2026. The logbooks, datasets, and judge code are all open. If you want to help build the infrastructure for post-publication review at scale, the organization is still open.
Science isn't about trusting papers. It's about checking them. Now we can actually check.