Do products built on the same base model fail on the same people and cases?

Research question

Among deployed generative AI products that are built on the same base model (for example, customer-service assistants, writing tools or screening tools from different firms), are errors and harmful outputs more strongly correlated across products, by case and by the group affected, than among products built on different base models?

Why it matters

The infrastructure account predicts that homogenization creates single points of failure and that flaws are inherited by every adapted model. Those are its distinctive observables, the ones the normal-technology and cultural-technology accounts do not imply, but the sources read for this debate state them without measuring them in deployment. Earlier work tested outcome homogenization on benchmarks with shared components, not across live products.

The debate lists this among what we still do not know: Are failures and biases correlated across products built on the same base models, as the infrastructure account predicts?

Evidence that motivates it

1. Bommasani and colleagues named “foundation models” for their “critically central yet incomplete character,” argued that their significance lies in “emergence and homogenization,” warned that homogenization “creates single points of failure,” and predicted that the barrier to entry for developing them “will continue to rise.” (LD01-SL-C28 v1.0)
   Foundation models are said to create single points of failure whose flaws are inherited downstream.
   Cites: Bommasani et al. (2021), “On the Opportunities and Risks…”
2. Van der Vlist, Helmond and Ferrari describe “Big AI” as the structural convergence of AI and Big Tech, arguing that AI is “an actual technology stack encompassing infrastructure, models, applications” and that Amazon, Microsoft and Google have integrated all three into their cloud offerings. (LD01-SL-C29 v1.0)
   AI described as a stack integrated into three firms' cloud offerings.
   Cites: van der Vlist et al. (2024), “Big AI: Cloud…”
3. Luitse and Denkena examine large language models as “infrastructural components for AI research and development,” arguing that they are intertwined with big technology companies’ business models and, using GPT-3, describing “corporate efforts to commodify” them “through paid API access and exclusive licensing.” (LD01-SL-C30 v1.0)
   Language models as infrastructural components sold through metered access.
   Cites: Luitse & Denkena (2021), “The great Transformer…”

Claims are shown as they read on the day this pack was built; claim pages keep every earlier version.

A possible design

Identify sets of deployed products in one domain whose base model is disclosed or can be identified, half sharing a base model and half not. Submit a common, pre-registered set of test cases, including cases from different demographic groups, through each product's public interface, repeat over time, and estimate the correlation of errors across products by shared versus different base model, controlling for task and vendor. Where possible, compare with the base model's own errors on the same cases.

Data needed

Product inventories with base-model disclosures; a test-case set with group labels; repeated access to public interfaces; terms-of-service review for testing.

Feasibility

A PhD project. Medium; subscription and API fees, and coder time for judging outputs. Base models are often undisclosed, and vendors change them; record versions and dates. Testing through public interfaces must respect terms of service, which may limit the sample.

Has this been done?

Partly done already: related work covers some of this question (first-pass screen of 16 October 2026; not yet checked in depth).

No empirical study was found that measures correlated failures across independently deployed generative AI products built on one base model. Nearest work measures outcome homogenization on benchmarks or models it formally.

Where the search could not reach. Screened by one web search, arXiv record lookups and the LD-01 source scan on the date shown; industry audits and model-card disclosures were not searched. Absence of a hit is not proof that the work does not exist.

Before building a project on it, run your own search in a scholarly database such as Scopus, Web of Science, Google Scholar or SSRN.

Nearest work found

• Bommasani, Creel, Kumar, Jurafsky & Liang, "Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?", arXiv:2211.13972 (2022) Judged from an abstract, summary or search listing only
  Abstract read. Formalizes outcome homogenization and tests it on benchmarks with shared training data and adapted foundation models, not on deployed products.
• Raghavan, "Competition and Diversity in Generative AI," arXiv:2412.08610 (2024) Judged from an abstract, summary or search listing only
  Abstract read. A game-theoretic model in which competition can reduce monoculture; no measurement across products.

References

Sources behind the motivating claims:

• Bommasani et al., “On the Opportunities and Risks of Foundation Models,” arXiv:2108.07258, 2021 (SRC-SL-BOMMASANI-ET-AL-ARXIV-2108-07258, original)
• van der Vlist, Helmond & Ferrari, “Big AI: Cloud infrastructure dependence and the industrialisation of artificial intelligence,” Big Data & Society, 2024 (SRC-SL-VAN-DER-VLIST-HELMOND-FERRARI-BDS-2024, original)
• Luitse & Denkena, “The great Transformer: Examining the role of large language models in the political economy of AI,” Big Data & Society, 2021 (SRC-SL-LUITSE-DENKENA-BDS-2021, original)

BibTeX

The first entry cites this opportunity; then its nearest work; then the sources above. Nearest work and sources are recorded as reference strings, so each goes in whole as the title; add authors and year from the original before you cite it.

@misc{sociolens_ro_ld01_03_v1,
  author = {{SocioLens project}},
  title = {Do products built on the same base model fail on the same people and cases?},
  year = {2026},
  date = {2026-10-16},
  howpublished = {\url{https://sociolens-web.pages.dev/opportunities/ro-ld01-03/v1/}},
  note = {Research opportunity RO-LD01-03, version v1}
}

@misc{ro_ld01_03_nearest_1,
  title = {Bommasani, Creel, Kumar, Jurafsky \& Liang, "Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?", arXiv:2211.13972 (2022)},
  url = {https://arxiv.org/abs/2211.13972},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{ro_ld01_03_nearest_2,
  title = {Raghavan, "Competition and Diversity in Generative AI," arXiv:2412.08610 (2024)},
  url = {https://arxiv.org/abs/2412.08610},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated; Judged from an abstract, summary or search listing only}
}

@misc{src_sl_bommasani_et_al_arxiv_2108_07258,
  title = {Bommasani et al., “On the Opportunities and Risks of Foundation Models,” arXiv:2108.07258, 2021},
  url = {https://arxiv.org/abs/2108.07258},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_van_der_vlist_helmond_ferrari_bds_2024,
  title = {van der Vlist, Helmond \& Ferrari, “Big AI: Cloud infrastructure dependence and the industrialisation of artificial intelligence,” Big Data \& Society, 2024},
  url = {https://doi.org/10.1177/20539517241232630},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}

@misc{src_sl_luitse_denkena_bds_2021,
  title = {Luitse \& Denkena, “The great Transformer: Examining the role of large language models in the political economy of AI,” Big Data \& Society, 2021},
  url = {https://doi.org/10.1177/20539517211047734},
  note = {Reference as recorded by SocioLens; authors, title and year are not separated}
}