The Overlooked Frontier of AI Risk

Summary: The governance of training data is fast emerging as one of the most critical - and least understood - dimensions of artificial intelligence (AI) risk. While policymakers and practitioners have long focused on model outputs, fairness, and explainability, recent reports from the Organization for Economic Co-operation and Development (OECD) show that the foundations of AI models - the data they are trained on - pose equal if not greater governance challenges. As AI adoption accelerates, understanding how data are collected, labeled, and curated is becoming a defining issue for responsible AI practice.

Why Training Data Matters for AI Safety, Fairness and Competition

Training data determine the scope, limitations, and potential harms of AI systems. Bias in datasets can propagate into biased decisions, privacy lapses, and discriminatory outcomes. For example, under-representation of certain demographics in facial recognition datasets has been linked to error rates up to 34 percent higher for darker-skinned women compared with lighter-skinned men¹. Beyond fairness, the provenance and licensing of training data affect intellectual property (IP) rights and competition - especially as generative models scrape vast amounts of web content. The result is a complex interplay of ethical, legal, and market-level risks.

Policymakers are beginning to respond. The EU’s Artificial Intelligence Act (AI Act) requires that training data used in high-risk systems be relevant, representative, and free from errors “as far as possible” (Article 10). Similarly, the U.S. NIST AI Risk Management Framework (RMF) stresses dataset integrity and documentation as core components of trustworthy AI.

OECD Findings on Data-Collection Mechanisms

In 2025, the OECD published a detailed mapping of data-collection mechanisms driving AI training². The report identifies four dominant pathways:

1. Web-scraped data, gathered at scale with limited consent or oversight;

2. Licensed data, obtained through formal partnerships or open-data initiatives;

3. Synthetic data, generated to supplement or balance existing datasets; and

4. Proprietary in-house data, collected directly by firms for specific applications.

Each pathway carries distinct governance challenges. Web-scraping raises copyright and privacy concerns; licensed data rely on contractual clarity and metadata provenance; synthetic data demand validation to ensure they replicate realistic distributions without amplifying bias. OECD analysts conclude that few organizations maintain sufficient transparency about the origins of their training data, creating risks of non-compliance under data-protection and competition law.

Policy Gaps: Privacy, Intellectual Property, and Transparency

Despite broad recognition of data-governance risks, regulatory mechanisms remain fragmented. Privacy frameworks like the EU GDPR or California’s CPRA were not designed with machine-learning datasets in mind. They struggle to address composite data that blend personal and non-personal information. IP law also lags behind technological realities: copyright exemptions for “text and data mining” vary across jurisdictions, creating uncertainty for developers using open-source or web-scraped content.

Transparency is another weak point. While the EU AI Act introduces documentation requirements, it stops short of mandating full public disclosure of datasets. In practice, many developers withhold dataset details to protect trade secrets, leaving regulators and civil-society groups unable to audit for bias or provenance.

What Organizations Should Do Now: Audits, Documentation, and Data-Governance Frameworks

Responsible AI practice begins with visibility. Organizations should:

• Map data lineage - document the sources, licenses, and transformations applied to all training data;

• Conduct regular dataset audits - assess for representativeness, bias, and regulatory exposure;

• Implement data-provenance metadata - adopt standards such as the Data Provenance and Lineage schema recommended by ISO/IEC 5259-2 (currently in early adoption across industry);

• Build interdisciplinary review boards - involving legal, ethical, and technical stakeholders to assess dataset composition before model training.

Leaders can look to frameworks such as the NIST AI RMF, ISO/IEC 42001 (AI Management Systems), and OECD’s Principles on AI as scaffolds for data-governance programs. These resources emphasize proportional controls - ensuring governance effort aligns with risk level and model impact.

Future Regulatory Trends: Dataset Disclosure, Data Trusts, and Model-Governance Regimes

Regulators are already signaling a shift toward stricter dataset accountability. The European Commission is considering secondary legislation under the AI Act that could require “dataset cards” or structured metadata disclosures similar to model cards. In the UK, the Centre for Data Ethics and Innovation (CDEI) has promoted “data trusts” as mechanisms for collective data stewardship, balancing innovation with individual rights.

In the longer term, transparency expectations will likely converge across domains—training data for foundation models, medical AI, and public-sector systems alike. Companies that establish rigorous governance today will not only reduce compliance risk but also strengthen trust and competitiveness. As the OECD report underscores, data are not merely inputs to AI—they are the substrate of responsibility itself.

References