In science, is it the wording that converges or the ideas?

Research question

In scientific papers and proposals written with language models, how much of the measured rise in similarity is shared wording and style, and how much is shared research questions, methods and claims, when the two are measured separately and validated against expert judgement?

Why it matters

Most homogenization measures in writing are text-similarity scores, which cannot tell shared wording from shared ideas. One study of code finds convergence in syntax but not in approach, and the surface-not-substance position (P5) depends on this distinction. If ideas converge as well as wording, the narrowing concern is about knowledge, not style; if only wording converges, much of the measured homogenization is cosmetic.

The debate lists this among what we still do not know: When writing converges, is it the wording or the ideas?

Evidence that motivates it

1. Kaggle competition code from 2019 to mid-2026 has become more alike in its literal syntax, but the author finds no evidence of homogenization in the approach or intent of the code. (LD09-SL-C33 v1.0)
   Kaggle code converged in syntax, with no evidence of homogenization in approach or intent.
   Cites: Burtch (2026), “The Hitchhiker’s Guide to Monoculture: AI…”
2. By February 2024 an estimated 17.5% of sentences in computer-science abstracts on arXiv were substantially modified by language models; the share was higher in papers closer to their nearest peer, a proxy for more crowded research areas (22.2% against 14.7% for less similar papers). (LD09-SL-C09 v1.0)
   Model-modified sentences were more common in papers closer to their nearest peer.
   Cites: Liang et al. (2024), “Mapping the Increasing Use of LLMs…”
3. Re-analysing a published set of five brainstorming experiments, the authors found ChatGPT-aided idea sets less diverse than comparison sets in all five (significantly in four), and lower on all of their alternative diversity metrics, with 37 of 45 comparisons statistically significant; in one task 6% of ChatGPT-only ideas were unique against 100% of human-only ideas. (LD09-SL-C12 v1.0)
   Idea sets were less diverse on most alternative measures in a brainstorming re-analysis.
   Cites: Meincke et al., “ChatGPT Decreases Idea Diversity in…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Sample pairs of papers within subfields before and after 2023, matched on topic. For each paper, extract a structured record of research question, method, data and main claim, checked by domain experts, and separately a style representation from the text with content words masked. Estimate change in idea-level and style-level similarity as a function of estimated model use, and validate both measures against blind expert ratings of how alike pairs of papers are in substance.

Data needed

Full texts in two or three fields; domain experts for extraction checks and ratings; a distribution-level estimator of model-modified text.

Feasibility

One PhD chapter or paper. Low to medium; mostly expert time. Model-based extraction of ideas can itself homogenize; experts should check a random sample and the analysis should be repeated with a second extraction model.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 15 October 2026; not yet checked in depth).

Studies measure stylistic markers or overall semantic similarity of abstracts, and one measures proposal distinctiveness, but none was found that validates separate idea-level and style-level measures of convergence against expert judgement.

Where the search could not reach. Screened by two web searches and arXiv record lookups on the date shown; scientometrics journals were not searched directly. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Qian, Wen, Furnas, Bai, Shao & Wang, "The Rise of Large Language Models and the Direction and Impact of US Federal Research Funding," arXiv:2601.15485 (2026) Judged from an abstract, summary or search listing only
  Abstract read. Lower semantic distinctiveness of proposals with more model involvement; one embedding measure that mixes wording and substance.
• Kousha & Thelwall, multi-database study of model-associated vocabulary, arXiv:2509.09596 (2025) Judged from an abstract, summary or search listing only
  Abstract read. Vocabulary markers only.
• Raccuglia, master's thesis on homogenization of academic abstracts, University of Padua (2025) Judged from an abstract, summary or search listing only
  Seen in search results only; not read. Reported to find style changes without significant homogenization of lexicon and overall meaning in about 60,000 abstracts.
• Burtch, Kaggle code study (LD09-SL-C33) Judged from an abstract, summary or search listing only
  Abstract read for this debate. Separates syntax from approach, in code only.
• Safari & Babaei, "Research Novelty in Information Systems Journals After ChatGPT: Differences Across Institutional Language Contexts," arXiv:2603.22510 (2026) Judged from an abstract, summary or search listing only
  Abstract read. One embedding-based novelty measure of titles and abstracts, which mixes wording and substance.

References

Sources behind the motivating claims:

• Burtch, “The Hitchhiker’s Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics,” arXiv:2607.13077, 2026 (SRC-SL-BURTCH-MONOCULTURE-ARXIV-2607-13077, original)
• Liang, Zhang, Wu, Lepp, Ji, Zhao et al., “Mapping the Increasing Use of LLMs in Scientific Papers,” arXiv:2404.01268, April 2024 (SRC-SL-LIANG-ET-AL-ARXIV-2404-01268, original)
• Meincke, Nave & Terwiesch, “ChatGPT Decreases Idea Diversity in Brainstorming,” author manuscript, The Wharton School, University of Pennsylvania (SRC-SL-MEINCKE-NAVE-TERWIESCH-CHATGPT-IDEA-DIVERSITY, no checked link to the original yet)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld09_06_v1,
  author = {{SocioLens project}},
  title = {In science, is it the wording that converges or the ideas?},
  year = {2026},
  date = {2026-10-15},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld09-06/v1/}},
  note = {Research opportunity RO-LD09-06, version v1}
}

@misc{ro_ld09_06_nearest_1,
  title = {Qian, Wen, Furnas, Bai, Shao \& Wang, "The Rise of Large Language Models and the Direction and Impact of US Federal Research Funding," arXiv:2601.15485 (2026)},
  url = {https://arxiv.org/abs/2601.15485},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld09_06_nearest_2,
  title = {Kousha \& Thelwall, multi-database study of model-associated vocabulary, arXiv:2509.09596 (2025)},
  url = {https://arxiv.org/abs/2509.09596},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld09_06_nearest_3,
  title = {Raccuglia, master's thesis on homogenization of academic abstracts, University of Padua (2025)},
  url = {https://thesis.unipd.it/handle/20.500.12608/84777},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld09_06_nearest_4,
  title = {Burtch, Kaggle code study (LD09-SL-C33)},
  url = {https://arxiv.org/abs/2607.13077},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld09_06_nearest_5,
  title = {Safari \& Babaei, "Research Novelty in Information Systems Journals After ChatGPT: Differences Across Institutional Language Contexts," arXiv:2603.22510 (2026)},
  url = {https://arxiv.org/abs/2603.22510},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_burtch_monoculture_arxiv_2607_13077,
  title = {Burtch, “The Hitchhiker’s Guide to Monoculture: AI Homogenizes Syntax, Not (Necessarily) Semantics,” arXiv:2607.13077, 2026},
  url = {https://arxiv.org/abs/2607.13077},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_liang_et_al_arxiv_2404_01268,
  title = {Liang, Zhang, Wu, Lepp, Ji, Zhao et al., “Mapping the Increasing Use of LLMs in Scientific Papers,” arXiv:2404.01268, April 2024},
  url = {https://arxiv.org/abs/2404.01268},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_meincke_nave_terwiesch_chatgpt_idea_diversity,
  title = {Meincke, Nave \& Terwiesch, “ChatGPT Decreases Idea Diversity in Brainstorming,” author manuscript, The Wharton School, University of Pennsylvania},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}