Claim LD06-SL-C20 (v1.0)

On a benchmark of 1,000 transcripts from 61 datasets, frontier models distinguished evaluation transcripts from real use well above chance (Gemini 2.5 Pro: AUC 0.83, against 0.92 for two of the paper’s authors rating every transcript, a baseline the authors treat as an approximate upper bound), more so in agentic settings than in chat.

Research opportunities this claim motivates

What this claim leaves open for research. Each is a hypothesis about a gap, not a finding.

How much does knowing it is tested change a model's measured safety?Frontier models distinguish evaluation transcripts from real use with AUC around 0.83, below the 0.92 of two of the study's authors, which the paper treats as an approximate upper bound.

Provenance and status

Claim type: ESTIMATE (as printed: “ESTIMATE”)
Publication status: ELIGIBLE (PREPRINT) (as printed: “ELIGIBLE WITH PREPRINT QUALIFIER”)
Verification state: CONTENT_VERIFIED_PRIMARY (PREPRINT) (as printed: “CONTENT VERIFIED PREPRINT”; verified by: UNKNOWN)
Does not license: How much this changes models’ behaviour in any evaluation
Source (field 6, as printed): Needham, Edkins, Pimpale, Bartsch & Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836 (v3), July 2025
Source type (field 7): PREPRINT (as printed: “Preprint”)
Institutional position: Research programme and evaluation organization (MATS and Apollo Research)
Maturity (claim evidence / publication maturity): EMERGING (as printed: “EMERGING”)
Recorded date: 2026-10-15
Currentness / correction: This is the current version. No correction or concern relation is recorded on its Sources.

Version

Current version v1.0, resolved through the explicit current pointer CLAIM--LD06-SL-C20.

Versions:v1.0 · current

Pinned Sources (field 6, version-specific)

“Needham, Edkins, Pimpale, Bartsch & Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836 (v3), July 2025” → SRC-SL-NEEDHAM-ET-AL-EVALUATION-AWARENESS-ARXIV-2505-23836--v1.0 — Needham, Edkins, Pimpale, Bartsch & Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836 (v3), July 2025; class PREPRINT (explicit public provenance pin)

All 25 canonical Claim fields (raw values and normalized values)

#	Canonical field	Canonical value (verbatim)	Normalized (Claim Value Vocabulary v1.0.2)
1	CLAIM ID	LD06-SL-C20	—
2	PUBLIC-FACING CLAIM	On a benchmark of 1,000 transcripts from 61 datasets, frontier models distinguished evaluation transcripts from real use well above chance (Gemini 2.5 Pro: AUC 0.83, against 0.92 for two of the paper’s authors rating every transcript, a baseline the authors treat as an approximate upper bound), more so in agentic settings than in chat.	—
3	CLAIM TYPE	ESTIMATE	base_types: ESTIMATE
4	RESEARCH STRAND	B — Leaderboards, judges and disclosure	—
5	MATURITY	EMERGING	states: EMERGING
6	SOURCE	Needham, Edkins, Pimpale, Bartsch & Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836 (v3), July 2025	source_ids: SRC-SL-NEEDHAM-ET-AL-EVALUATION-AWARENESS-ARXIV-2505-23836
7	SOURCE TYPE	Preprint	classes: PREPRINT
8	INSTITUTIONAL POSITION	Research programme and evaluation organization (MATS and Apollo Research)	—
9	SCOPE	Chat and agentic transcripts from evaluations and from deployment	—
10	DESIGN	Classification benchmark of evaluation versus deployment transcripts	—
11	SAMPLE / DATASET	1,000 transcripts from 61 datasets	—
12	MODEL / TOOL / VERSION	Gemini 2.5 Pro, Claude 3.7 Sonnet, GPT-4.1 and others	—
13	COMPARATOR	Human raters	—
14	MAIN RESULT	Gemini 2.5 Pro AUC 0.83; Claude 3.7 Sonnet and GPT-4.1 about 0.82–0.83; two authors as human baseline 0.92 (an approximate upper bound, not typical human capability); models far above chance at identifying what an evaluation tests	—
15	INFERENTIAL TARGET	Whether models can tell they are being tested	—
16	DOES NOT LICENSE	How much this changes models’ behaviour in any evaluation	—
17	VERIFICATION STATE	CONTENT VERIFIED PREPRINT	state: CONTENT_VERIFIED_PRIMARY · qualifiers: PREPRINT · verified_by: UNKNOWN
18	LIMITATIONS	Recognition, not behaviour change, was measured; the human baseline is two of the authors, not a general rater sample; Apollo sells evaluations	—
19	CONTRADICTORY / SUPPORTING SOURCES	Behavioural evidence in LD06-SL-C21	—
20	GEOGRAPHY / LANGUAGE	Not checked / English	—
21	DISTRIBUTIONAL LEDGER	Not analysed in this claim	—
22	R01 / FIELD-MAP LINKS	LD-06, D4b	—
23	WHAT WOULD CHANGE THIS CLAIM?	Evidence that recognition does not alter evaluated behaviour	—
24	PUBLICATION STATUS	ELIGIBLE WITH PREPRINT QUALIFIER	state: ELIGIBLE · qualifiers: PREPRINT
25	RECORDED DATE	2026-10-15	—

Field 22 references resolved against Field Map v0.2: LD-06 (LIVING_DEBATE); D4b (SUB_AREA)

Cited by Living Debate versions

LD-06 v0.1

Research-of-record provenance

Materialized by sociolens-materializer 0.6.0 at 2026-09-28T20:00:00.000Z; normalization contract SocioLens — Claim Value Vocabulary v1.0.2 — Pre-Spike Materialization Contract (v1.0.2). Content seal SHA-256 9ad85721ab996acec8ed5f69ee6d016b4dc0bfaa00de95092660077f5ed4e1fd.

Repository-authored inputs (not Drive exports)

SocioLens claim — LD06-SL-C20 v1.0 (v1.0) — written in the SocioLens repository by Project Lead (Claude, AI) on 2026-10-15; no Drive identity exists. Authority: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts. Block “ENTRY 20 — LD06-SL-C20” SHA-256 6285c7c78860f41ecea6ae4cd1e121be26877968d868c4cd241fee393020e425; fixture fixtures/authored/claims/LD06-SL-C20--v1.0.

Builder decisions (not Drive data)

VERSION = v1.0 — basis: Claim version taken from the repository-authored claim fixture version (SocioLens claim — LD06-SL-C20 v1.0); the first version of a new Claim of the LD-06 debate
CURRENT_POINTER = true — basis: repository-authored claim "SocioLens claim — LD06-SL-C20 v1.0" (Project Lead (Claude, AI), 2026-10-15); basis: AGENTS.md "Authority" (the Project Lead decides research content within the research-integrity rules) and the Owner's choice in the SocioLens project chat on 15 October 2026 to build the LD-06 debate; a new Claim, first version, written from the project's LD-06 source notes, checked against full texts or official abstracts
FIELD_6_SOURCE_REFERENCE = SRC-SL-NEEDHAM-ET-AL-EVALUATION-AWARENESS-ARXIV-2505-23836--v1.0 — basis: field 6 of ENTRY 20 — LD06-SL-C20 cites "Needham, Edkins, Pimpale, Bartsch & Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836 (v3), July 2025"; the only field-6 segment of this ClaimVersion's repository-authored ledger block.
SOURCE_PROVENANCE_PIN = SRC-SL-NEEDHAM-ET-AL-EVALUATION-AWARENESS-ARXIV-2505-23836--v1.0 — basis: explicit version-pinned public provenance of this ClaimVersion (Amendment v1.2 §6): the pinned field-6 Source record "Needham, Edkins, Pimpale, Bartsch & Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836 (v3), July 2025".