Research opportunity · The evaluation apparatus: Can benchmarks, leaderboards and audits carry the weight placed on them?
Do benchmark scores predict what models achieve in real deployments?
Question
Across many language models, do scores on public benchmarks predict measured outcomes when the same models are used in real work, such as code that maintainers merge in live repositories, support tickets resolved or error rates in use, and are the benchmarks calibrated to those outcomes or only correct in rank?
Wrong or missing validation targetA PhD projectLow to medium costPartly addressed
Why it matters
Benchmark scores are used to choose, buy and regulate models on the assumption that they predict performance in use. The checks found so far compare one coding benchmark with maintainers' review of the benchmark's own tasks, or one model in one trial, and both find the score running ahead of accepted work. If scores predict field outcomes in rank but not in level, they can still guide choice between models but not claims about readiness; if they do not predict rank, the benchmark-versus-use position (P5) is right and much of the evaluation economy rests on an untested proxy.
The debate lists this among what we still do not know: Do benchmark scores across many models predict measured outcomes in deployment, in any domain?
Evidence that motivates it
- When four maintainers of three repositories reviewed 296 AI-written pull requests that had passed SWE-bench Verified’s automated tests, roughly half would not have been merged; on average, maintainer merge rates, scored relative to the 68% at which the same maintainers would merge the human-written patches actually merged, ran about 24 percentage points below the benchmark scores.
Maintainers would not have merged about half of the test-passing SWE-bench Verified patches they reviewed; merge rates, scored relative to the maintainers' 68% golden-patch baseline, ran about 24 points below benchmark scores. - In interviews with 19 people who use or decided against using AI benchmarks, academics generally saw public benchmarks as suitable for tracking research progress, while in product and policy work benchmarks, even internally built ones, were often found inadequate for substantive decisions.
Product and policy practitioners often found benchmarks inadequate for substantive decisions. - The International AI Safety Report 2026 finds an “evaluation gap”: performance on pre-deployment tests does not reliably predict real-world utility or risk, models “now regularly identify evaluation prompts as tests”, and although 12 companies published or updated frontier safety frameworks in 2025, there are no standardised external audits of their compliance.
The International AI Safety Report 2026 names an evaluation gap between pre-deployment test performance and real-world utility or risk. - An audit of agentic benchmarks found flaws in task setup or scoring, for example insufficient test cases in SWE-bench Verified and empty responses counted as successes in TAU-bench, that can under- or overestimate agents’ performance by up to 100% in relative terms.
Flaws in agentic benchmarks' scoring can misstate performance by up to 100% in relative terms.
A possible design
Assemble a panel of models for which both public benchmark scores and field outcomes exist. One route uses public records of AI-agent pull requests in open-source repositories (merged or not, with the agent and model identified) and relates each model's merge rate, adjusted for repository and task, to its SWE-bench and other coding scores. A second route partners with an organization that has switched models over time in one workflow and logs outcomes. Estimate rank correlation and calibration, with pre-registered hypotheses and out-of-sample checks on models released after the analysis plan.
Data needed
Public pull-request datasets that identify the agent and model; benchmark scores from independent runs where possible, not only developer reports; for the second route, an organization's outcome logs across model versions.
Feasibility
A PhD project. Low to medium; public repository data is free, and compute for independent benchmark runs is modest, but the organizational partner route needs data access. Model identity in public pull requests is often incomplete; selection into which tasks people give to agents must be modelled, not assumed away.
Has this been done?
Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).
No study was found that links many models' public benchmark scores to field outcomes of the same models in real deployments and estimates rank correlation or calibration. Existing work compares a benchmark with maintainer review of the benchmark's own tasks, a benchmark with other benchmarks or downstream test tasks, or a lab's pre-release simulation with its own post-release data; field studies of agent pull requests report merge rates per tool without tying them to benchmark scores.
How strong this evidence is
- Who searched and checked: .
- What was read: 5 of the 6 nearest works below were judged from an abstract, summary or search listing only; each is marked. Not every entry says how much of the work was read.
- Where the search could not reach: Screened by web search (4 queries) and one arXiv query on the date shown; economics and management venues, clinical prospective-validation studies and enterprise case studies were not searched. Absence of a hit is not proof that the work does not exist.
- Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.
Nearest work found
- P. Whitfill et al., "Many SWE-bench-Passing PRs Would Not Be Merged into Main," METR research note (10 March 2026)
Full text read for the LD-06 debate (LD06-SL-C12). Hypothetical maintainer review of benchmark patches, not outcomes from live deployment. - Gevers & Daelemans, "Benchmarking the Benchmarks: Testing the Predictive Validity of Commonsense Benchmarks," arXiv:2608.03340 (2026) Judged from an abstract, summary or search listing only
Abstract read. 23 models, four commonsense benchmarks and eight downstream tasks with rank comparison; the criterion is downstream test tasks, not field outcomes. - Williams, Sheahan, Raymond, Korbak et al., "Predicting LLM Safety Before Release by Simulating Deployment," arXiv:2607.07184 (2026) Judged from an abstract, summary or search listing only
Abstract read. Replays earlier deployment conversations to predict post-release misbehaviour across four releases of one lab; validates against deployment, but within one developer and not for public benchmark scores. - Patel, El Maghraoui et al., "Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents," arXiv:2606.19704 (2026) Judged from an abstract, summary or search listing only
Abstract read. Aggregates fourteen implementation studies of one industrial-agent benchmark and argues for defining predictive validity as in-sample to out-of-sample rank correlation; calls existing evidence too thin to confirm. - Mazloomzadeh, Morovati & Khomh, "How Do AI Coding Agents Contribute to Software Development? An Empirical Study of Agentic Pull Requests," arXiv:2607.21832 (2026) Judged from an abstract, summary or search listing only
Abstract read. Field merge outcomes per agent tool, with no link to benchmark scores; a possible outcome source. - Ruan, Maddison & Hashimoto, "Observational Scaling Laws and the Predictability of Language Model Performance," arXiv:2405.10938 (2024) Judged from an abstract, summary or search listing only
Abstract read. Predicts benchmarks from other benchmarks across about 100 models; benchmark-to-benchmark only.
Search queries used
benchmark scores predict real-world deployment outcomes LLM predictive validity field dataAI coding agent pull requests merge rate GitHub across models compared with SWE-bench scores empirical studycorrelation between Chatbot Arena or benchmark rank and real-world usage outcomes production A/B test LLMs studymodel switching enterprise logs LLM benchmark rank versus customer outcomes resolution rate across models study 2026arXiv: abs:benchmark AND abs:"predictive validity"
Provenance
RO-LD06-01 v1. Drafted by Claude (SocioLens Project Lead), AI-drafted from the LD-06 debate on 2026-10-15. Not yet reviewed. Not yet reviewed by a domain expert. This is a research hypothesis, not a finding.