Do benchmark scores predict what models achieve in real deployments?

Research question

Across many language models, do scores on public benchmarks predict measured outcomes when the same models are used in real work, such as code that maintainers merge in live repositories, support tickets resolved or error rates in use, and are the benchmarks calibrated to those outcomes or only correct in rank?

Why it matters

Benchmark scores are used to choose, buy and regulate models on the assumption that they predict performance in use. The checks found so far compare one coding benchmark with maintainers' review of the benchmark's own tasks, or one model in one trial, and both find the score running ahead of accepted work. If scores predict field outcomes in rank but not in level, they can still guide choice between models but not claims about readiness; if they do not predict rank, the benchmark-versus-use position (P5) is right and much of the evaluation economy rests on an untested proxy.

The debate lists this among what we still do not know: Do benchmark scores across many models predict measured outcomes in deployment, in any domain?

Evidence that motivates it

1. When four maintainers of three repositories reviewed 296 AI-written pull requests that had passed SWE-bench Verified’s automated tests, roughly half would not have been merged; on average, maintainer merge rates, scored relative to the 68% at which the same maintainers would merge the human-written patches actually merged, ran about 24 percentage points below the benchmark scores. (LD06-SL-C12 v1.0)
   Maintainers would not have merged about half of the test-passing SWE-bench Verified patches they reviewed; merge rates, scored relative to the maintainers' 68% golden-patch baseline, ran about 24 points below benchmark scores.
   Cites: Whitfill et al. (2026), “Many SWE-bench-Passing PRs Would…”
2. In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions. (LD06-SL-C13 v1.0)
   Product and policy practitioners often found benchmarks inadequate for substantive decisions.
   Cites: Hardy et al. (2024), “More than Marketing? On the…”
3. The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance. (LD06-SL-C14 v1.0)
   The International AI Safety Report 2026 names an evaluation gap between pre-deployment test performance and real-world utility or risk.
   Cites: Bengio et al. (2026), “International AI Safety Report 2026”
4. An audit of agentic benchmarks found flaws in task setup or scoring, for example insufficient test cases in SWE-bench Verified and empty responses counted as successes in TAU-bench, that can under- or overestimate agents’ performance by up to 100% in relative terms. (LD06-SL-C08 v1.0)
   Flaws in agentic benchmarks' scoring can misstate performance by up to 100% in relative terms.
   Cites: Zhu et al. (2025), “Establishing Best Practices for…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Assemble a panel of models for which both public benchmark scores and field outcomes exist. One route uses public records of AI-agent pull requests in open-source repositories (merged or not, with the agent and model identified) and relates each model's merge rate, adjusted for repository and task, to its SWE-bench and other coding scores. A second route partners with an organization that has switched models over time in one workflow and logs outcomes. Estimate rank correlation and calibration, with pre-registered hypotheses and out-of-sample checks on models released after the analysis plan.

Data needed

Public pull-request datasets that identify the agent and model; benchmark scores from independent runs where possible, not only developer reports; for the second route, an organization's outcome logs across model versions.

Feasibility

A PhD project. Low to medium; public repository data is free, and compute for independent benchmark runs is modest, but the organizational partner route needs data access. Model identity in public pull requests is often incomplete; selection into which tasks people give to agents must be modelled, not assumed away.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).

No study was found that links many models' public benchmark scores to field outcomes of the same models in real deployments and estimates rank correlation or calibration. Existing work compares a benchmark with maintainer review of the benchmark's own tasks, a benchmark with other benchmarks or downstream test tasks, or a lab's pre-release simulation with its own post-release data; field studies of agent pull requests report merge rates per tool without tying them to benchmark scores.

Where the search could not reach. Screened by web search (4 queries) and one arXiv query on the date shown; economics and management venues, clinical prospective-validation studies and enterprise case studies were not searched. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• P. Whitfill et al., "Many SWE-bench-Passing PRs Would Not Be Merged into Main," METR research note (10 March 2026)
  Full text read for the LD-06 debate (LD06-SL-C12). Hypothetical maintainer review of benchmark patches, not outcomes from live deployment.
• Gevers & Daelemans, "Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks," arXiv:2608.03340 (2026) Judged from an abstract, summary or search listing only
  Abstract read. 23 models, four commonsense benchmarks and eight downstream tasks with rank comparison; the criterion is downstream test tasks, not field outcomes.
• Williams, Sheahan, Raymond, Korbak et al., "Predicting LLM Safety Before Release by Simulating Deployment," arXiv:2607.07184 (2026) Judged from an abstract, summary or search listing only
  Abstract read. Replays earlier deployment conversations to predict post-release misbehaviour across four releases of one lab; validates against deployment, but within one developer and not for public benchmark scores.
• Patel, El Maghraoui et al., "Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents," arXiv:2606.19704 (2026) Judged from an abstract, summary or search listing only
  Abstract read. Aggregates fourteen implementation studies of one industrial-agent benchmark and argues for defining predictive validity as in-sample to out-of-sample rank correlation; calls existing evidence too thin to confirm.
• Mazloomzadeh, Morovati & Khomh, "How Do AI Coding Agents Contribute to Software Development? An Empirical Study of Agentic Pull Requests," arXiv:2607.21832 (2026) Judged from an abstract, summary or search listing only
  Abstract read. Field merge outcomes per agent tool, with no link to benchmark scores; a possible outcome source.
• Ruan, Maddison & Hashimoto, "Observational Scaling Laws and the Predictability of Language Model Performance," arXiv:2405.10938 (2024) Judged from an abstract, summary or search listing only
  Abstract read. Predicts benchmarks from other benchmarks across about 100 models; benchmark-to-benchmark only.

References

Sources behind the motivating claims:

• Whitfill, Wu, Becker & Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026 (SRC-SL-WHITFILL-WU-BECKER-RUSH-METR-2026-03-10, original)
• Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein & Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024 (SRC-SL-HARDY-ET-AL-MORE-THAN-MARKETING-ARXIV-2412-05520, original)
• Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026 (SRC-SL-INTERNATIONAL-AI-SAFETY-REPORT-2026, original)
• Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025 (SRC-SL-ZHU-ET-AL-AGENTIC-BENCHMARK-CHECKLIST-ARXIV-2507-02825, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld06_01_v1,
  author = {{SocioLens project}},
  title = {Do benchmark scores predict what models achieve in real deployments?},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-01/v1/}},
  note = {Research opportunity RO-LD06-01, version v1}
}

@misc{ro_ld06_01_nearest_1,
  title = {P. Whitfill et al., "Many SWE-bench-Passing PRs Would Not Be Merged into Main," METR research note (10 March 2026)},
  url = {https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld06_01_nearest_2,
  title = {Gevers \& Daelemans, "Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks," arXiv:2608.03340 (2026)},
  url = {https://arxiv.org/abs/2608.03340},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_01_nearest_3,
  title = {Williams, Sheahan, Raymond, Korbak et al., "Predicting LLM Safety Before Release by Simulating Deployment," arXiv:2607.07184 (2026)},
  url = {https://arxiv.org/abs/2607.07184},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_01_nearest_4,
  title = {Patel, El Maghraoui et al., "Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents," arXiv:2606.19704 (2026)},
  url = {https://arxiv.org/abs/2606.19704},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_01_nearest_5,
  title = {Mazloomzadeh, Morovati \& Khomh, "How Do AI Coding Agents Contribute to Software Development? An Empirical Study of Agentic Pull Requests," arXiv:2607.21832 (2026)},
  url = {https://arxiv.org/abs/2607.21832},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_01_nearest_6,
  title = {Ruan, Maddison \& Hashimoto, "Observational Scaling Laws and the Predictability of Language Model Performance," arXiv:2405.10938 (2024)},
  url = {https://arxiv.org/abs/2405.10938},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_whitfill_wu_becker_rush_metr_2026_03_10,
  title = {Whitfill, Wu, Becker \& Rush, “Many SWE-bench-Passing PRs Would Not Be Merged into Main,” METR research note, 10 March 2026},
  url = {https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_hardy_et_al_more_than_marketing_arxiv_2412_05520,
  title = {Hardy, Reuel, Jafari Meimandi, Soder, Griffith, Asmar, Koyejo, Bernstein \& Kochenderfer, “More than Marketing? On the Information Value of AI Benchmarks for Practitioners,” arXiv:2412.05520, December 2024},
  url = {https://arxiv.org/abs/2412.05520},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_international_ai_safety_report_2026,
  title = {Bengio, Clare, Prunkl, Andriushchenko, Bucknall, Murray et al., “International AI Safety Report 2026,” arXiv:2602.21012, February 2026},
  url = {https://arxiv.org/abs/2602.21012},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_zhu_et_al_agentic_benchmark_checklist_arxiv_2507_02825,
  title = {Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui et al., “Establishing Best Practices for Building Rigorous Agentic Benchmarks,” arXiv:2507.02825 (v5), 2025},
  url = {https://arxiv.org/abs/2507.02825v5},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}