Walk-forward backtest · 2003–2023 · Research leaderboard

Beating the Median, Not the Best

Machine Learning versus Venture Capitalists in Startup Selection

A public-data machine-learning screen beats the median venture capitalist at picking startups — not the best.

Anatole Chapelier15 November 2026Not fund returns · Not investment advice

Leaderboard · 3-year follow-on financing · 2003–2009

Cumulative equivalent hits (k = 50). Every contestant is normalized to a 50-deal book each vintage.

Not fund returns · walk-forward backtest · ai-vs-vc

How to read this

Each line is a contestant in a walk-forward race. Every vintage, everyone “picks” a book the same size as the official top-50 portfolio. A hit is a pick that later raised a follow-on round within three years — or, with the Exit toggle, that had an IPO or acquisition within five.

This window is the official comparison against reconstructed VC books.

Hits: Cumulative equivalent successes, 50 × vintage hit rate, added year by year. All contestants use the same k.
Hit rate: Running cumulative hit rate: those equivalent hits divided by 50 and by the number of vintages so far.
GBM: Our histogram gradient-boosting model, scored on the same 7,200-deal sample as the LLMs.
VC books: Reconstructed first-round portfolios, median and 75th percentile, on the full universe. Not a model top-k.
LLMs: Numeric P(success) on anonymized, point-in-time cards. Grok 4.7 is a 682-deal subsample.

This is a screening metric, not a dollar P&L and not a fund multiple. A higher hit rate can still lose money on entry price.

Leaderboard

Official period averages from the LLM-as-investor benchmark: unweighted mean, across vintages, of the scaled top-50-equivalent hit rate and of per-vintage AUC. Intervals are 95% percentile bootstrap confidence intervals. “YES” means the model beats that benchmark on hit rate after Holm correction at 5%. GBM here is scored on the same 7,200-deal sample as the language models.

Follow-on 2003–2009. YES means the hit-rate comparison survives Holm correction at 5%.
Contestant	AUC	Top-50 hit rate	Beats median VC	Beats p75 VC	Beats GBM
Claude Sonnet 5.5	0.660 0.637–0.682	82.0% 74.4%–91.2%	YES	YES	no
GPT-6.1 Sol	0.661 0.638–0.683	76.8% 64.2%–85.5%	YES	no	no
LLM ensemble	0.654 0.631–0.676	73.8% 63.6%–83.8%	YES	no	no
GBM (v3 full)	0.674 0.652–0.697	67.5% 55.7%–80.1%	YES	no	—
DeepSeek V3.2	0.621 0.600–0.645	66.0% 41.7%–66.5%	no	no	no
Grok 4.7Stratified subsample (682 main deals).	0.549 0.472–0.629	57.1% 39.9%–68.6%	no	no	no
Qwen3-235B	0.592 0.569–0.613	42.6% 26.9%–51.7%	no	no	no
Llama 4 Maverick	0.616 0.594–0.641	36.4% 24.5%–50.8%	no	no	no
Median VCOfficial reconstructed book (full universe).	—	49.0%	—	—	—
Top-quartile VCOfficial reconstructed book (full universe).	—	64.2%	—	—	—
Base rateSample label mean.	—	40.3%	—	—	—
Permuted-label placebo	0.537 0.513–0.562	50.0% 39.7%–64.2%	—	—	—

Holm and Benjamini–Hochberg corrections are applied across the pre-declared family of tests (see the paper). VC books are unavailable after 2015, so those columns are blank on 2016–2023. Grok 4.7 is a stratified subsample; every Grok comparison with GBM is on that exact subset. The primary full-universe backtest — the one behind the title — is summarized below. It is a slightly different sample from the 7,200-deal LLM arm, so the GBM hit rates are not identical.

Full universe · Follow-on 2003–2009

68.7%

GBM top-50 · median VC 49.0% · p75 64.2%

Full universe · Follow-on 2010–2015

86.8%

GBM top-50 · median VC 65.3% · p75 79.0%

Full universe · Exits 2005–2009

27.0%

GBM top-50 · median VC 19.3% · p75 32.1%

LLM sample · Claude, 2003–2009 FO

82.0%

Beats median and p75 VC after Holm · not GBM

Key findings

5 results that survive the footnotes. The recognition caveat should travel with any LLM headline.

01 The median, not the best

Public-data GBM beats the median VC as a screen.

On the full-universe walk-forward backtest (v3_full HistGradientBoosting, top-50 books, after 448 outcome-date corrections), GBM’s 3-year follow-on hit rate is 68.7% in 2003–2009 versus 49.0% for the median VC book and 64.2% for the 75th-percentile book, and 86.8% versus 65.3% / 79.0% in 2010–2015. On 5-year IPO-or-acquisition exits in 2005–2009 it beats the median (27.0% vs 19.3%) but not the 75th percentile (27.0% vs 32.1%). That is the claim in the title.

02 LLMs look strong — until you ask if they know the company

Frontier models beat the median VC on hit rate; much of the edge is recognition.

On anonymized first-round cards for 7,200 walk-forward deals, Claude Sonnet 5.5 posts an 82.0% scaled top-50 follow-on hit rate in 2003–2009 (significantly above both the median and the 75th-percentile VC after Holm correction) and GPT-6.1 Sol posts 76.8% (above the median). The five-model ensemble is at 73.8%. A recognition test, however, shows GPT named 11.2% of asked companies and Claude 5.7%. Drop those firms and Claude’s hit rate on the asked 2003–2009 slice falls from 74.4% to 62.4%; GPT falls from 69.8% to 47.1% — the median VC book.

03 Not better than the tabular model

No LLM significantly beats GBM after multiple-testing correction.

On the same companies, Claude’s follow-on AUC in 2003–2009 is 0.660 versus GBM’s 0.674; GPT is 0.661. Hit-rate point estimates can favour Claude, but after Holm and Benjamini–Hochberg correction no LLM significantly beats GBM on any primary period. After 2015, LLM AUCs sit near chance (Claude 0.536, GPT 0.532) while GBM remains at 0.629. Adding the LLM score as a GBM feature lifts AUC from 0.67 to 0.72 in 2003–2009 and does nothing in 2016–2023.

04 Blind cards shrink the older-vintage edge

With names, cities, and descriptions removed, no LLM hit-rate test survives Holm.

On a 500-deal pretest, no model recognized a firm from the stricter cards (0.0% vs 11.2% GPT and 5.7% Claude on the original cards). On matched companies, 2003–2009 follow-on AUCs fall: GPT 0.643 to 0.588 (p=0.002), DeepSeek 0.621 to 0.589 (p=0.008), the open-model ensemble 0.632 to 0.603 (p=0.004). Claude’s matched hit rate is 65.7% (n=517) versus 71.4% on original cards. After Holm correction, no blind-card LLM beat the median VC, the 75th-percentile VC, or GBM. Grok was not rerun.

05 This is not a fund return

A better screen is not a TVPI, and the best books still win on exits.

Hit rate is a screening metric: whether a first-round pick later raised again or exited. It is not a net-of-fee multiple. Among 48 early-stage positions in 20 public-fund SEC schedules that later leave the filing, the median last-mark multiple is 0.67× — and disappearance from a filing is not realized proceeds. Use this page as a research leaderboard, not as investment advice.

Methods, in brief

Walk-forward backtest with point-in-time features only. Models train on earlier vintages and are tested on the next year’s first rounds. The LLM arm scores anonymized cards (names, URLs, and post-round information withheld) at temperature 0 and reports a numeric P(success). Top-k is scaled to the official v3 top-50 quantile. Uncertainty is a within-vintage bootstrap (1,000 replicates); primary tests use Holm and BH corrections.

Universe: 305,010 companies assembled from Y Combinator, historical Crunchbase snapshots, SEC Form D, Fundraising Fox, Field–Ritter IPO dates, and Wikidata.
448 documented outcome-date corrections; the effect on the six primary tests is under 0.6 percentage points.
Primary model: histogram gradient boosting with the v3_full point-in-time feature set. Logistic regression and random forest were also run.
Benchmarks: reconstructed VC first-round books (median and 75th percentile), the sample/universe base rate, and a permuted-label placebo.
LLM sample: 7,200 first-round deals, 2003–2023, stratified by vintage and base rate, plus 2,000 recent deals from 2024–2026.
Models scored: Claude Sonnet 5.5, GPT-6.1 Sol, Llama 4 Maverick, Qwen3-235B-A22B-Instruct-2507, DeepSeek V3.2 (full sample); Grok 4.7 on a 682-deal subsample.
A stricter de-identified prompt (‘blind cards’) was scored on 15 November 2026: Llama, Qwen, and DeepSeek on the full 7,200, Claude and GPT on a 1,545-deal subsample. Toggle Blind cards to compare. Grok was not rerun.

Paper

Chapelier, A. (2026). Beating the Median, Not the Best: Machine Learning versus Venture Capitalists in Startup Selection. Prepared as an English-language arXiv preprint. The PDF linked here is the current working draft.

Read the PDF Dataset notes

Data

The research extract covers 305,010 companies and the associated first-round books, walk-forward scores, SEC fund-schedule panel, and 59,000 LLM replies. It is distributed as a separate archive (about 293 MB compressed), not bundled in this page. The numbers on the chart and leaderboard are computed from those files; this site ships only the aggregated series in data.json.

Corrected company and round tables, plus a log of the 448 outcome-date fixes and the residual uncorrected flags.
Walk-forward backtest v3 after those corrections, including reconstructed VC books and the primary GBM, logistic regression, and random forest runs.
LLM-as-investor scores, recognition flags, and official period metrics used on this page.
SEC fund-schedule panel, rebuilt from the raw filings: 1,259 reports, 591 positions, 48 early-stage positions that later leave the schedule.

Limitations

These are screening hit rates, not realized fund returns, TVPI, or IRR. A higher hit rate can still lose money on entry price.
VC benchmarks are reconstructed books from public first-round appearances, not the internal memos or opportunity set of those firms.
LLM cards withheld names, but models can still recognize firms from city, amounts, investor counts, or descriptions. On original cards, GPT named 11.2% of asked companies and Claude 5.7%. A stricter blind-card recognition test on 500 deals was 0% for every model; ranking edges on older vintages still shrink.
Grok 4.7 is a stratified subsample, not the full 7,200 deals, and was not given the recognition test.
Reconstructed VC books are unavailable after 2015 in the official file. Five-year exit results for the LLM arm stop in 2009.
Claude, GPT, and Grok cannot yet be tested on post-cutoff deals with a full 3- or 5-year outcome window.
Some DeepSeek vintage hit rates are unstable because of score ties and a small scaled k; prefer AUC when the two disagree.
The 448 date corrections and the residual uncorrected flags are documented; they do not overturn the primary conclusions, but the labels are still incomplete public data.