The Race — Building Democracy in the Age of AI. People race toward a shared finish line.
The Race

Policy tool · See record

Independent testing and evaluation

What the tool does

Have people outside the supplier’s development team test an AI system’s performance and risks. Give evaluators enough access, clear methods and the ability to report adverse findings so purchasing and oversight do not depend entirely on vendor assurances.

Current status

A developing institutional practice. Public evaluation arrangements exist, and audit laws provide additional models. Shared testing capacity is not automatically independent: evaluator selection, access, conflicts and publication rights determine how much scrutiny is possible.

The proposed Treasury/GAO arrangement below is not verified enacted or operating.

Where it has been used

The GSA–NIST evaluation partnership

Added by

NIST’s March 2026 agreement with GSA establishes technical and methodological support for evaluating AI used in federal procurement. It builds shared public evaluation capacity; it does not establish that every government AI system has completed an independent audit.

Read more about this case →

California’s independent-oversight programme

Added by

California’s September 18, 2026 executive account describes implementation work on independent AI oversight under SB813 and AB1405. It concerns building an oversight arrangement. The account does not establish that all contemplated independent audits have been completed.

Read more about this case →

A proposed independent review of an AI tax pilot

Added on

H.R.9501, reported September 8, 2026, proposes an AI tax-fraud pilot run by the US Treasury. The pilot would begin within 180 days of enactment and operate for 18–24 months. An independent Government Accountability Office report would follow within 180 days after completion, examining fraud detected, money recovered and tool accuracy. The proposal does not establish that a pilot has launched, that an evaluation has succeeded or that revenue has been recovered.

Read more about this case →

TIGTA’s review of IRS AI practices

Added on

A September 24, 2026 Treasury Inspector General for Tax Administration report reviewed documentation for selected IRS uses of AI. It found missing assessment and testing records. The review did not test the accuracy of the underlying data or establish that the IRS had completed remediation; it is documentary oversight rather than validation of a model.

Read more about this case →

New Mexico’s proposed safety regime

Added on

New Mexico’s October 1 frontier-safety summary describes audits and safety commitments in proposed 2027 legislation. It is a proposed oversight design, not an operating audit requirement. The summary does not establish completed independent audits or the full scope of auditor access and enforcement.

Read more about this case →

New York City’s AI validation bill

Added on

The City Council’s official record now confirms introduction on October 8 of Introduction 1100-2026, previously identified as T2026-2602. The bill remains in committee. The October 5 hearing and October 8 introduction did not enact it.

The proposal would prohibit marketing, selling or deploying an AI model in the city without third-party validation and a technical ability for a human operator to shut it down. Validation would address matters including performance, disparate impact, privacy and safety. NYC Cyber Command would establish implementation rules and validator qualifications; the proposal includes civil penalties, including $25,000 per instance for specified failures or falsified validation.

Read more about this case →

Cantwell’s proposed frontier-AI framework

Added on

Senator Maria Cantwell published a governance framework October 7. It calls for measurable catastrophic-risk standards, independent pre-release audits, disclosure and incident reporting, accountability and whistleblower protections, public defensive capacity, and human oversight and recourse. It also addresses children, workers and international coordination.

This is an outline of desired legislation, not an enacted system or a newly enforceable obligation. Its value is that it describes several concrete protections that could replace the vague instruction to “regulate AI.” It does not yet specify all the institutions, thresholds, remedies or implementation details needed to make those protections work.

Read more about this case →

Pennsylvania’s frontier-AI safety bill

Added on

Pennsylvania House Bill 2829 was introduced and referred to committee October 5. It targets developers with more than $500 million in annual revenue and frontier models meeting a specified training-computation threshold. If enacted, its disclosure and filing system would begin January 1, 2028.

The bill calls for a documented safety framework, annual independent audits, auditor access to unredacted information, incident reports and whistleblower protections. The Pennsylvania Emergency Management Agency would administer the system, and the attorney general could seek penalties and injunctions. Its catastrophic-risk definition is narrow: it focuses on specified very large harms, rather than creating comprehensive protection against every harmful AI use.

Read more about this case →

ICO supervision of AI developers

Added on

On October 8, the UK Information Commissioner’s Office (ICO) reported privacy changes secured through its supervision of ten major AI developers: Amazon, Anthropic, Apple, Cohere, DeepSeek, Google, Meta, Microsoft, OpenAI and Stability AI. These are companies developing many models, not ten individual large language models. Some changes are completed; others remain commitments. The work applies existing UK data-protection duties to model training and use, including explaining whose personal data is used, identifying a lawful basis and helping people exercise access and objection rights.

For example, the supervision account documents changes to privacy notices and explanations by Apple, Cohere and OpenAI, and reviews of the justification for using personal data by other developers. On the same day, the ICO opened a public call for evidence on AI agents, which can take actions for users. It closes November 20 and asks about transparency, accountability, security and automated decisions before future guidance and a statutory code. The consultation is open now; the future code is not yet adopted. The underlying data-protection law is compulsory even where firms cooperate without a new enforcement order. Regulator privacy supervision is external scrutiny, not a newly established independent model benchmark.

Read more about this case →

Britain’s AI security-practice agenda

Added on

Britain’s AI Security Institute has published the agenda promised in the minister’s October 12 speech. It asks developers and technical organizations to collaborate on practices covering the entire development process, including training, internal deployment and evaluation. Its priorities include risk documentation, monitoring, incident reporting, evaluator access/resources/publication, and measurement of dangerous capabilities.

The institute explicitly says it is not mandating requirements. It proposes developing and sharing voluntary practices first, with mature practices potentially passing into international standards organizations later. No adopted compulsory standard, enforceable incident-reporting duty or new public right of access is established by this publication. It supplies more detail than the speech, while leaving authority and participation terms to future arrangements.

Read more about this case →

External evaluators’ publication rights

Added on

Reuters reported October 13 that Anthropic and OpenAI have sought influence over what embedded evaluators publish. Anthropic distinguishes established evaluators from new firms for which it reserves report-approval rights; OpenAI says it seeks confidentiality redactions rather than approval of full reports. The reporting describes negotiations and differing accounts, not a settled industry rule.

The underlying tension is confirmed in public records. Anthropic’s September partnership announcement says access/reporting standards and funding arrangements remain unsettled. OpenAI’s principles combine editorial independence with confidentiality and intellectual-property protection. In an earlier incident investigation, METR describes six days of on-site work, redactions, and edits following OpenAI feedback, while stating that no additional material important to its conclusions was removed except where noted. This is not proof that all assessments were censored or independent.

Read more about this case →

Scientists’ international AI security proposal

Added on

On October 13, 2026, an international group including Geoffrey Hinton and Yoshua Bengio called for stronger containment of AI systems capable of cyber operations. The proposal would initially isolate such systems from external networks, develop technical standards and create an international body for incident information and oversight.

An air gap means disconnecting a test environment from the internet or other networks. The letter asks governments and developers to cooperate; its publication does not establish an operating regulator, an adopted reporting obligation or a funded international institution.

Read more about this case →

Massachusetts’ proposed evaluator rights

Added on

October 13 reporting describes continued negotiations over AI provisions in Massachusetts' Mass Wins legislation. The relevant Senate amendment, S.3228, dates to July 24; the governor subsequently urged inclusion of safety and transparency measures. This is pending legislation, not an October law taking effect. Coverage would require a frontier developer and its affiliates to exceed $500 million in annual revenue, with covered models crossing the bill's exceptionally large training-computation threshold (10^26 operations, including subsequent modifications).

The would require large frontier developers to obtain independent evaluations at least every 120 days and after specified risk reports. Evaluators would receive reasonably necessary materials and access to the most capable models, could question developers, and would publish a public report within 30 days of delivery. Security and confidentiality restrictions and permitted redactions remain. Developer payment is allowed under stated independence safeguards where another funding source has not been established.

Read more about this case →

METR’s embedded Anthropic evaluation

Added on

Anthropic announced on October 15, 2026 that six METR employees had company hardware, office space and internal access under enforceable agreements. METR can publish assessments without Anthropic’s general approval. An October 16 clarification states that Anthropic ultimately determines whether information falls within predefined redaction categories. METR retains specified rights to describe the access and information it received; it has no general veto over redactions. These are reported contractual terms rather than publicly inspected agreements or statutory supervision.

Read more about this case →

ByteDance’s internal safety team

Added on

ByteDance’s Seed unit announced on October 14, with reporting October 15, that it had established an Alignment & Preparedness team and planned an initial global recruitment of ten researchers. Its programme includes safety evaluations, adversarial testing, monitoring, alignment training and external research relationships. The announcement does not identify a published budget, independent appointment process or protected publication rights.

The institutional distinction matters: a developer can build useful safety expertise while retaining control over what is tested, what is disclosed and what happens after a bad result. External collaboration is an intention, not evidence of METR-like access or an independent regulator. Company account; Reuters reporting carried by GMA.

Read more about this case →

Source snapshot: 2026-10-17. Individual event and verification dates remain as recorded above. Inclusion alone does not assign a score.