Scorecard · United States
Anthropic clarifies who controls redactions in METR’s embedded evaluation
View the linked event assessment →
Anthropic added an October 16 clarification to its October 15 announcement about six METR employees working inside the company. It now explicitly says Anthropic ultimately decides whether information falls within categories it can require METR to redact. METR retains specified rights to describe the access and information it received. The arrangement still permits publication without Anthropic’s general approval; it does not give METR final authority over confidentiality classifications. Current primary account ↗.
For The Race, this qualifies yesterday’s proposed award. Actual internal access and a contractual publication route distribute some scrutiny outside the developer. Developer control over redactions limits that independence. These are reported enforceable agreements, not publicly inspected contracts or statutory supervision. The separate Accenture partnership remains at a less developed access/publication stage.
Approved disposition: Retain the existing Dispersion +1 award, dated October 15, with this narrower explanation; Leaning dispersion / Opens, bounded; lanes 2/1/6. This corrects the October 16 item 1 account, whose award has now been recorded; no second award. Keep it context-only for TR-019 if scored; update the existing independent-evaluation Tool in accountability-tools and EP-20261014-A. Check METR’s publication/redaction experience, contract duration and any dispute mechanism October 23. No standing trend reassessment yet.
Follow-up: October 23: METR contract duration, redaction/publication experience and dispute mechanism; October 30 existing continuity/first assessment check retained.
Current development (November 6): On November 5 the external evaluator METR reported that most members of its project team believe Anthropic was improving model capabilities on well-specified tasks in October at about twice its rate from March 2024 to October 2025. METR reviewed internal evidence and progress measures and had access to Anthropic staff. It reports substantial uncertainty and disagreement within its team. The comparison is an informed assessment for one lab and class of tasks; it does not isolate how much AI caused or show that other labs progress at the same rate. METR describes contractual publication routes and limits on what it could disclose. Assessment and access terms ↗.
The October 15 grant of protected access, METR-ANTHROPIC-20261015, already received Dispersion +1. This publication shows that an outside team could publish a substantive result and disclose its limits. It is evidence about the operation of the mechanism already scored.
Additional source perspective: Independent testing and evaluation
Read the canonical Policy Tool
What the tool does
Have people outside the supplier’s development team test an AI system’s performance and risks. Give evaluators enough access, clear methods and the ability to report adverse findings so purchasing and oversight do not depend entirely on vendor assurances.
Current status
A developing institutional practice. Public evaluation arrangements exist, and audit laws provide additional models. Shared testing capacity is not automatically independent: evaluator selection, access, conflicts and publication rights determine how much scrutiny is possible.
The proposed Treasury/GAO arrangement below is not verified enacted or operating.
Where it has been used
- United States: NIST’s agreement with GSA ↗ establishes methodological support for procurement evaluation.
- California: the September 2026 independent-oversight account ↗ describes implementation work for SB813 and AB1405; this does not prove completed independent audits.
- United States — proposed example, not verified use: H.R. 9501 reported September 8, 2026 ↗ would require Treasury to start an AI tax-fraud pilot within 180 days of enactment, operate it for 18–24 months, and obtain an independent GAO report within 180 days after completion on fraud detected, money actually recovered and tool accuracy. The latest official action verified October4 is September8; later enactment/launch is unverified. Committee report ↗; CBO estimate ↗. No new IRS authority, successful evaluation or substantial revenue gain is demonstrated. Related proposal: AI Tax Integrity Act; stable event
ai-tax-integrity-act-hr9501-pilot, supporting caseapproved-d1004-09. Added October4 under daily editorial approval; existing Policy Tool identity retained.
- October 5 dated example / qualification: D1005-09, September24 TIGTA report: documentary oversight of selected IRS AI uses. It establishes missing assessment/testing records; it did not test underlying-data accuracy or verify remediation. IRS November commitment is checked December1. This example does not claim a validated model.
New Mexico — proposed, not in force: the October1 frontier-safety summary ↗ describes audits and safety commitments for intended2027 legislation. Full bill text, evidence access and enforcement remain held; no completed independent audit is established. Case D1002-05a.
New York City — proposed, not enacted: T2026-2602 would require non-affiliate validation and shutdown capability. Developers engage validators; final qualification, access, coverage and remedies determine independence. October8 proceedings are future, not passage. Draft docket ↗; case.
October 8 — Cantwell outlines what frontier-AI regulation could require: Senator Maria Cantwell published a governance framework October 7. It calls for measurable catastrophic-risk standards, independent pre-release audits, disclosure and incident reporting, accountability and whistleblower protections, public defensive capacity, and human oversight and recourse. It also addresses children, workers and international coordination.
This is an outline of desired legislation, not an enacted system or a newly enforceable obligation. Its value is that it describes several concrete protections that could replace the vague instruction to “regulate AI.” It does not yet specify all the institutions, thresholds, remedies or implementation details needed to make those protections work. Case. Primary/source account ↗.
October 8 — Pennsylvania introduces a frontier-AI safety bill with audits and enforcement: Pennsylvania House Bill 2829 was introduced and referred to committee October 5. It targets developers with more than $500 million in annual revenue and frontier models meeting a specified training-computation threshold. If enacted, its disclosure and filing system would begin January 1, 2028.
The bill calls for a documented safety framework, annual independent audits, auditor access to unredacted information, incident reports and whistleblower protections. The Pennsylvania Emergency Management Agency would administer the system, and the attorney general could seek penalties and injunctions. Its catastrophic-risk definition is narrow: it focuses on specified very large harms, rather than creating comprehensive protection against every harmful AI use. Case. Primary/source account ↗.
October 9 — Britain’s data regulator reports changes by AI developers and opens an agents consultation: On October 8, the UK Information Commissioner’s Office (ICO) reported privacy changes secured through its supervision of ten major AI developers: Amazon, Anthropic, Apple, Cohere, DeepSeek, Google, Meta, Microsoft, OpenAI and Stability AI. These are companies developing many models, not ten individual large language models. Some changes are completed; others remain commitments. The work applies existing UK data-protection duties to model training and use, including explaining whose personal data is used, identifying a lawful basis and helping people exercise access and objection rights.
For example, the supervision account documents changes to privacy notices and explanations by Apple, Cohere and OpenAI, and reviews of the justification for using personal data by other developers. On the same day, the ICO opened a public call for evidence on AI agents, which can take actions for users. It closes November 20 and asks about transparency, accountability, security and automated decisions before future guidance and a statutory code. The consultation is open now; the future code is not yet adopted. The underlying data-protection law is compulsory even where firms cooperate without a new enforcement order. Regulator privacy supervision is external scrutiny, not a newly established independent model benchmark. Case. Source ↗.
October 9 — New York City formally introduces its AI validation and shutdown bill: The City Council’s official record now confirms introduction on October 8 of Introduction 1100-2026, previously identified as T2026-2602. The bill remains in committee. The October 5 hearing and October 8 introduction did not enact it.
The proposal would prohibit marketing, selling or deploying an AI model in the city without third-party validation and a technical ability for a human operator to shut it down. Validation would address matters including performance, disparate impact, privacy and safety. NYC Cyber Command would establish implementation rules and validator qualifications; the proposal includes civil penalties, including $25,000 per instance for specified failures or falsified validation. Official bill history and summary ↗. Source findings are proposed, not an operative obligation. Case. Source ↗.
- October 12 evidence — Britain proposes stronger AI security reporting while expanding government testing and cyberdefence: Britain's October 12 announcement combines existing security work with proposed new obligations. AI minister Kanishka Narayan says the AI Security Institute is evaluating the latest frontier models, including Google's Gemini 4 Argon, and that two government pilots have helped identify and address more than 1,000 vulnerabilities. These are government-reported activities and results. The announcement was previewed during the reporting window; the official October 12 publication does not establish an exact release time. Official account ↗.
The proposed next step is to strengthen the Cyber Security and Resilience Bill and the wider security regime so serious AI incidents and near misses can be reported and answered. Government will develop response capability with the National Cyber Security Centre and the AI Security Institute, including preparation for systems escaping intended human control. Those reporting duties and arrangements are being developed; they are not enacted obligations or a completed national response service. A separate claim that vulnerability-monitoring response times fell from 50 to eight days should not be attributed entirely to the AI pilots. Minister's speech ↗.
October 13 example/limitation — Britain publishes the promised agenda for frontier-AI technical practice: Britain’s AI Security Institute has published the agenda promised in the minister’s October 12 speech. It asks developers and technical organizations to collaborate on practices covering the entire development process, including training, internal deployment and evaluation. Its priorities include risk documentation, monitoring, incident reporting, evaluator access/resources/publication, and measurement of dangerous capabilities.
The institute explicitly says it is not mandating requirements. It proposes developing and sharing voluntary practices first, with mature practices potentially passing into international standards organizations later. No adopted compulsory standard, enforceable incident-reporting duty or new public right of access is established by this publication. It supplies more detail than the speech, while leaving authority and participation terms to future arrangements.
Case.
October 13 example/limitation — External AI evaluators’ publication rights are still being negotiated: Reuters reported October 13 that Anthropic and OpenAI have sought influence over what embedded evaluators publish. Anthropic distinguishes established evaluators from new firms for which it reserves report-approval rights; OpenAI says it seeks confidentiality redactions rather than approval of full reports. The reporting describes negotiations and differing accounts, not a settled industry rule.
The underlying tension is confirmed in public records. Anthropic’s September partnership announcement ↗ says access/reporting standards and funding arrangements remain unsettled. OpenAI’s principles ↗ combine editorial independence with confidentiality and intellectual-property protection. In an earlier incident investigation ↗, METR describes six days of on-site work, redactions, and edits following OpenAI feedback, while stating that no additional material important to its conclusions was removed except where noted. This is not proof that all assessments were censored or independent.
Case.
October 13 example/limitation — Scientists propose international oversight of AI containment and incident reporting: On October 13, an international group including Geoffrey Hinton and Yoshua Bengio called for stronger containment of AI systems capable of cyber operations. Independent accounts agree on the proposal’s core: initially isolate such systems from external networks, create an international body to standardize containment and track failures, and report significant incidents for eventual public disclosure. Reporting gives differing signature totals, so the exact count is left unverified.
An “air gap” means disconnecting a test environment from the internet or other external networks. The proposed institution would provide shared reporting and oversight rather than leaving the definition of adequate security entirely to each laboratory. The letter is a request to governments and developers; no government has established or funded this body in the evidence reviewed. Claims about particular security incidents require their own incident records, rather than confirmation merely through signatures. Reuters ↗ and the Canadian Press ↗ corroborate the initiative.
Case.
October 14 example — Massachusetts' pending bill would give evaluators access and publication duties:
October 13 reporting describes continued negotiations over AI provisions in Massachusetts' Mass Wins legislation. The relevant Senate amendment, S.3228 ↗, dates to July 24; the governor subsequently urged inclusion of safety and transparency measures. This is pending legislation, not an October law taking effect. Coverage would require a frontier developer and its affiliates to exceed $500 million in annual revenue, with covered models crossing the bill's exceptionally large training-computation threshold (10^26 operations, including subsequent modifications).
The primary text ↗ would require large frontier developers to obtain independent evaluations at least every 120 days and after specified risk reports. Evaluators would receive reasonably necessary materials and access to the most capable models, could question developers, and would publish a public report within 30 days of delivery. Security and confidentiality restrictions and permitted redactions remain. Developer payment is allowed under stated independence safeguards where another funding source has not been established.
Massachusetts' pending bill would give evaluators access and publication duties.
October 16 current qualification: The October 15 Anthropic update establishes stronger access/publication terms for METR and describes other partnerships as having more limited rights. Do not attribute METR’s rights to Accenture. Retain the September Anthropic–Accenture Dispersion +1 and its historical rationale; the original access/funding/publication comparison remains open, with a research check October 30. Sources: https://www.anthropic.com/news/accenture-embedded-evaluation ↗ ; https://www.anthropic.com/news/an-update-on-independent-evaluators ↗ .
- METR begins working inside Anthropic with protected publication rights — 2026-10-15: Anthropic announced on October 15 that six employees of METR, the nonprofit AI evaluator, have received company hardware, office space and internal-system access broadly comparable to its own risk-assessment teams. The arrangement operates under enforceable agreements, according to Anthropic. It excludes categories including customer information, legal privilege and model architectures. Other METR employees do not receive this access.
METR can publish assessments without Anthropic’s approval. Anthropic can require redactions within predefined categories and ultimately decides whether information falls within those categories. METR retains specified rights to describe the access and information it received; a general right to dispute all redactions is not established. This does not give METR a deployment veto or government enforcement power. METR’s separately published funding policy rejects financial dependence on AI companies; the exact engagement’s duration and termination provisions are not public in the evidence inspected. Anthropic terms ↗; METR policy, October 14 ↗.
For The Race, the concrete change is an outsider’s continuing access and ability to publish adverse conclusions. This satisfies the existing embedded-evaluator Watchlist trigger more closely than an announcement of future cooperation. The September Anthropic–Accenture Dispersion +1 is a separate record: its September 18 announcement ↗ describes direct provider funding and unsettled operating details, while the new update says other partnerships have more limited rights. We should qualify that record’s current description rather than imply Accenture already has METR’s rights; retain its historical score while reviewing the original rationale.
Scorecard now, Dispersion +1, October 15; lanes 2/1/6; Leaning dispersion / Opens; medium-high confidence. Promote the existing implementation-watch branch, preserve the Accenture identity/history and open its historical-description comparison. Add METR as context only to TR-019, excluded from trend scoring counts; update the existing independent-audit Policy Tool in accountability-tools. Add evidence to the approved evaluation-governance editorial outline. Check contractual continuity and the first public assessment by October 30; no new trend score.
- ByteDance creates an internal frontier-model safety team — 2026-10-14: ByteDance’s Seed unit announced on October 14, with reporting October 15, that it had established an Alignment & Preparedness team and planned an initial global recruitment of ten researchers. Its programme includes safety evaluations, adversarial testing, monitoring, alignment training and external research relationships. The announcement does not identify a published budget, independent appointment process or protected publication rights.
The institutional distinction matters: a developer can build useful safety expertise while retaining control over what is tested, what is disclosed and what happens after a bad result. External collaboration is an intention, not evidence of METR-like access or an independent regulator. Company account ↗; Reuters reporting carried by GMA ↗.
Unscored evaluation-institution case, lanes 2/1/7, Undetermined / Unclear. Add TR-019 context and the evaluation editorial outline as the internal-governance comparison. No Specific Event, new Tool or generic Watchlist; reconsider on actual outside access, disclosure obligations or independent authority.
- Proposal source recovered, not adopted: The scientists’ original cybersecurity letter ↗ proposes containment, an international standards/incident body and eventual incident disclosure. It does not establish an operating institution. The current signature count must not replace the dated launch-day count.
October 17 approved examples/qualification:
METR-ANTHROPIC-20261015 — Anthropic clarifies who controls redactions in METR’s embedded evaluation ↗ — Operating contractual access, bounded independence: Anthropic ultimately determines redaction-category applicability; METR can make specified access/information statements, not exercise a general redaction veto.
Anthropic added an October 16 clarification to its October 15 announcement about six METR employees working inside the company. It now explicitly says Anthropic ultimately decides whether information falls within categories it can require METR to redact. METR retains specified rights to describe the access and information it received. The arrangement still permits publication without Anthropic’s general approval; it does not give METR final authority over confidentiality classifications. Current primary account ↗.
For The Race, this qualifies yesterday’s proposed award. Actual internal access and a contractual publication route distribute some scrutiny outside the developer. Developer control over redactions limits that independence. These are reported enforceable agreements, not publicly inspected contracts or statutory supervision. The separate Accenture partnership remains at a less developed access/publication stage.
Supporting sources
Source snapshot: 2026-11-06. Individual event and verification dates remain as recorded above. Inclusion alone does not assign a score.
