Who builds and checks China's compliance test banks for generative AI?

Research question

Under China's national standard GB/T 45654-2025, who builds the generated-content and refusal test banks that decide whether a generative AI service passes, how are the banks validated, and do providers' banks and third-party evaluators' banks give the same verdicts on the same models?

Why it matters

China has written evaluation into compliance as fixed pass rates on test banks, a model unlike the EU's. The validity critiques of benchmarks (errors, contamination, unclear constructs) apply with force if providers build the banks they are scored on. Almost all evaluation research concerns English-language benchmarks and US, UK and EU institutions, so a study of the Chinese regime would also correct the field's geography.

The debate lists this among what we still do not know: Who builds the test banks under China's standard, and how are they checked?

Evidence that motivates it

1. China’s national standard GB/T 45654-2025 for generative AI services sets a pass rate: at least 90% of sampled output must be free of 31 risk types; its reference appendix specifies a test bank of at least 2,000 questions, refusal of at least 95% of should-refuse questions, and refusal of no more than 5% of should-not-refuse questions. (LD06-SL-C25 v1.0)
   The standard requires a 90% qualified rate on sampled output and, in its reference appendix, test banks of at least 2,000 questions with refusal thresholds of 95% and 5%.
   Cites: State Administration for Market Regulation &…
2. Re-annotating 5,700 questions across all 57 subjects of MMLU, a widely used knowledge benchmark, researchers estimated that 6.49% of its questions contain errors, 57% of those analysed in the Virology subset, and that the errors change how models rank. (LD06-SL-C04 v1.0)
   Benchmark items contain errors that change model rankings, a risk for any fixed bank.
   Cites: Gema et al. (2025), “Are We Done with MMLU?”
3. An analysis of MMLU found that 28% of its questions require culturally sensitive knowledge and that 84.9% of its geography questions concern North America or Europe; model rankings change depending on whether the culturally sensitive questions are included, so machine-translated versions distort multilingual comparisons. (LD06-SL-C11 v1.0)
   Benchmark content reflects where it was made, which bears on comparing Chinese and Western safety testing.
   Cites: Singh et al. (2025), “Global MMLU: Understanding and…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

A comparative study combining document analysis of published evaluation datasets and reports from Chinese third-party evaluation institutions, standard-setting bodies and providers; interviews with staff of evaluation institutions and providers; and, where banks or samples are public, an item audit for errors and overlap and a comparison of verdicts on the same open-weight models.

Data needed

Chinese-language documents and datasets (including any state-built benchmark datasets); interviewees in evaluation institutions; open-weight Chinese models for verdict comparisons.

Feasibility

A PhD project. Low to medium; researcher time, translation and some compute. Needs Chinese-language skills and a Chinese-literature search first; access to providers' banks is unlikely, so the design should rest on public datasets and interviews. Political sensitivity of the risk categories may limit what interviewees say.

Has this been done?

The search found no study that does the core design (first-pass screen of 15 October 2026; not yet checked in depth).

No study was found, in English or in the few Chinese results that surfaced, of who builds the test banks the standard requires, how they are validated, or whether providers' and third-party evaluators' banks agree. Existing work is the standard and its translation, academic safety benchmarks aligned with Chinese regulation, and one state-built benchmark dataset seen only in search listings.

Where the search could not reach. Screened by web search (4 queries, one in Chinese) and one arXiv query on the date shown; CNKI, Wanfang and other Chinese databases could not be searched, so the absence of hits is especially weak evidence here. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• State Administration for Market Regulation & Standardization Administration of China, GB/T 45654-2025 (CSET English translation, 28 May 2026)
  Full text read for the LD-06 debate (LD06-SL-C25). The primary text; says nothing about how banks are built or checked in practice.
• CNCERT, "安全测评基准数据集1.0" (Safety evaluation benchmark dataset 1.0); publisher, date and URL not confirmed Judged from an abstract, summary or search listing only Some details not confirmed
  Judged from search listings; not read directly. Reportedly 3,000 questions covering the standard's 31 risk subcategories; a state-built bank and a key comparison object.
• Zhang, Gao, Hu et al., "ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models," arXiv:2410.18491 (2024) Judged from an abstract, summary or search listing only
  Abstract read. An academic benchmark aligned with Chinese content-moderation rules; does not study compliance banks.
• Zhang, Kodama, Wu, Chen, Zhu & Hong, "Frontier Safety Policies for AI Emergency Preparedness in China," arXiv:2511.05526 (2025) Judged from an abstract, summary or search listing only
  Abstract read. Policy analysis noting required pre-deployment assessments; does not study test banks.

References

Sources behind the motivating claims:

• State Administration for Market Regulation & Standardization Administration of China, GB/T 45654-2025, “Cybersecurity Technology — Basic Safety Requirements for Generative Artificial Intelligence Services,” issued 25 April 2025 (CSET English translation, 28 May 2026) (SRC-SL-GB-T-45654-2025-CSET-TRANSLATION, original)
• Gema, Leang, Hong et al., “Are We Done with MMLU?,” arXiv:2406.04127 (v3), January 2025 (SRC-SL-GEMA-ET-AL-ARXIV-2406-04127, original)
• Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025 (SRC-SL-SINGH-ET-AL-GLOBAL-MMLU-ARXIV-2412-03304, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld06_05_v1,
  author = {{SocioLens project}},
  title = {Who builds and checks China's compliance test banks for generative AI?},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld06-05/v1/}},
  note = {Research opportunity RO-LD06-05, version v1}
}

@misc{ro_ld06_05_nearest_1,
  title = {State Administration for Market Regulation \& Standardization Administration of China, GB/T 45654-2025 (CSET English translation, 28 May 2026)},
  url = {https://cset.georgetown.edu/publication/china-gen-ai-safety-standard-final/},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{ro_ld06_05_nearest_2,
  title = {CNCERT, "安全测评基准数据集1.0" (Safety evaluation benchmark dataset 1.0); publisher, date and URL not confirmed},
  url = {https://www.secrss.com/articles/83388},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only; Some details not confirmed}
}

@misc{ro_ld06_05_nearest_3,
  title = {Zhang, Gao, Hu et al., "ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models," arXiv:2410.18491 (2024)},
  url = {https://arxiv.org/abs/2410.18491},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld06_05_nearest_4,
  title = {Zhang, Kodama, Wu, Chen, Zhu \& Hong, "Frontier Safety Policies for AI Emergency Preparedness in China," arXiv:2511.05526 (2025)},
  url = {https://arxiv.org/abs/2511.05526},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_gb_t_45654_2025_cset_translation,
  title = {State Administration for Market Regulation \& Standardization Administration of China, GB/T 45654-2025, “Cybersecurity Technology — Basic Safety Requirements for Generative Artificial Intelligence Services,” issued 25 April 2025 (CSET English translation, 28 May 2026)},
  url = {https://cset.georgetown.edu/wp-content/uploads/t0661_gen_ai_safety_standard_final_EN.pdf},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_gema_et_al_arxiv_2406_04127,
  title = {Gema, Leang, Hong et al., “Are We Done with MMLU?,” arXiv:2406.04127 (v3), January 2025},
  url = {https://arxiv.org/abs/2406.04127v3},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_singh_et_al_global_mmlu_arxiv_2412_03304,
  title = {Singh, Romanou, Fourrier, Adelani, Ngui, Vila-Suero et al., “Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation,” arXiv:2412.03304 (v2), 2025},
  url = {https://arxiv.org/abs/2412.03304v2},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}