Eval companies win where a buyer needs a score it cannot make itself

The openings sit where a payer needs a score that platforms cannot produce

We checked possible eval companies against three tracks: money flows, forced demand and thin supply. The answer is a short list.

A good opening passes a two-part test. First, the score feeds a decision that someone already pays for. Examples are a procurement gate, an insurance premium, a lab's release decision or a supervisor's review. Second, the score needs something platforms cannot absorb: independence from the vendor, or ground truth from the real world.

Platforms absorb whatever runs on the customer's own traces. Datadog charges no separate fee for evaluations.^{[21][21] LLM Observability Datadog Primary} AWS bills $1.50 per 1,000 custom evaluations, and Microsoft bills tokens "with no surcharge".^{[23][23] AgentCore pricing AWS Primary[24][24] Foundry observability pricing Microsoft Azure Primary} A lab or cloud cannot credibly grade itself, so independence resists absorption. Real users, real application state and physical rollouts resist it too.

The three tracks point the same way.

• Money. 2026 rounds moved toward simulation, environments and certification. Exits still price most builder tools as features (next section).
• Demand. Buyers want tests that predict production, proof that shortens a sale, and inputs for insurers and supervisors.
• Gaps. Supply is thin where a test needs independence or real-world ground truth. It is dense where a test looks like software monitoring.

Investors fund eval startups modestly, and buyers pay feature prices for most of them

Venture money for eval companies is real but small next to lab-paid data and AI security. Exits show what buyers think a test is worth.

Eval-core startups raised about $1.5 billion in 22 months. We summed 45 disclosed equity rounds from January 2025 to 6 November 2026.^{[2][2] Tally of disclosed eval-core equity rounds, Jan 2025 to 6 Nov 2026 Ascending Labs Estimate} The total is a floor, because many seed rounds stay undisclosed. Arena alone took $450 million across three rounds, about 29%.^{[3][3] Series B announcement Arena Vendor[4][4] LMArena Series A SiliconANGLE Secondary} Without Arena and LangChain, 41 rounds share about $960 million, with a median near $15 million.

Headline "AI security" money is mostly not testing money. Momentum Cyber counts $7.5 billion of cyber financing in H1 2026, with 40% going to "AI security" firms.^{[5][5] Cybersecurity Market Review H1 2026 Momentum Cyber Secondary} That is about $3.0 billion, by our arithmetic. The largest of those rounds went to data security, security operations and AI attack tools, not to testing AI. Our tally finds about $0.35 billion of eval-core rounds in the same half-year.^{[2][2] Tally of disclosed eval-core equity rounds, Jan 2025 to 6 Nov 2026 Ascending Labs Estimate}

Earlier data shows the same split. One Crunchbase-based count found $8.5 billion raised by "AI security" startups in 2024–25. Only $414 million of it, under 5%, went to AI and LLM security proper (secondary).^{[7][7] AI security startups funding 2025 Software Strategies Blog Secondary}

Eval tools sell as features, so most exits stay modest. Dynatrace paid $915 million for Arize, about seven times Arize's funding.^{[8][8] Dynatrace to acquire Arize Dynatrace Vendor} CrowdStrike reportedly pays about $95 million for Dreadnode and about $400 million for Fiddler (WSJ via Dealroom).^{[9][9] CrowdStrike to buy Fiddler and Dreadnode for $495M Dealroom citing the Wall Street Journal Secondary} Check Point bought Deepchecks for about $15 million in enterprise value, close to its funding.^{[5][5] Cybersecurity Market Review H1 2026 Momentum Cyber Secondary[12][12] Check Point buys Deepchecks for up to $20M Dealroom citing Calcalist Estimate}

The large prices sit next to evaluation, not inside it. Palo Alto Networks paid $634.5 million for Protect AI, a model-security firm.^{[13][13] Form 10-K, fiscal 2025 Palo Alto Networks Primary} Cisco paid $400 million for Astrix, 16 times its trailing revenue, in identity security.^{[5][5] Cybersecurity Market Review H1 2026 Momentum Cyber Secondary} Business Insider reported talks of over $1.5 billion for Mechanize, as a licence plus hires (secondary).^{[14][14] Google may spend $1.5bn on Mechanize The Next Web Secondary}

Five kinds of buyer absorb eval startups.

• Security platforms buy red-teaming to fill a CISO suite: Cisco, Palo Alto, Zscaler, Check Point, Fortinet, A10 and CrowdStrike.^{[15][15] Agent security consolidation tracker Deepak Gupta Single source[9][9] CrowdStrike to buy Fiddler and Dreadnode for $495M Dealroom citing the Wall Street Journal Secondary}
• Observability and data platforms buy evals as telemetry features: CoreWeave, ClickHouse, Cisco, Dynatrace, Databricks and ServiceNow.^{[19][19] ClickHouse acquires Langfuse SiliconANGLE Secondary[20][20] Galileo acquisition Cisco Primary[8][8] Dynatrace to acquire Arize Dynatrace Vendor[1][1] Company directory (256 entries) Ascending Labs Primary}
• Model labs buy talent and fold testing into their platforms. OpenAI bought Promptfoo, and Google reportedly licensed Mechanize.^{[11][11] OpenAI acquires Promptfoo Yahoo Finance Secondary[14][14] Google may spend $1.5bn on Mechanize The Next Web Secondary}
• Vertical apps and roll-ups buy evals as an internal capability: Harvey, Beacon, Anaconda and Mintlify.^{[15][15] Agent security consolidation tracker Deepak Gupta Single source[1][1] Company directory (256 entries) Ascending Labs Primary}
• Expert-data firms buy environments and graders: Mercor bought Deeptune.^{[18][18] Mercor to acquire Deeptune Mercor Primary}

Some segments have seen no strategic buyer yet. No platform has bought a simulation, independent benchmarking, certification or dangerous-capability testing firm. The partial exceptions are a pivot (Patronus), a roll-up (Haize Labs) and a talent deal (Mechanize).^{[15][15] Agent security consolidation tracker Deepak Gupta Single source[14][14] Google may spend $1.5bn on Mechanize The Next Web Secondary} Buyers absorb what already runs on their data plane: traces or network traffic.

Investors price lab-paid revenue like staffing. In our sample, lab-paid labour and data trade at about 3 to 10 times revenue. Momentum Cyber puts AI security at 15 to 20 times or more.^{[5][5] Cybersecurity Market Review H1 2026 Momentum Cyber Secondary} Builder evals and leaderboards carry 23 to 100 times. Strategic buyers then pay a fraction of those marks (our arithmetic on reported revenue).

Eight crowded markets are the wrong place to start a new eval company

A red ocean has many sellers, a free substitute and buyers that bundle the product. Eight eval markets meet that test in November 2026.

Crowding alone does not rule a market out. Free substitutes and platform owners do. Sometimes the customer's cloud, monitoring vendor or model lab ships the same check at no extra fee. A startup then competes with zero. Its best exit is then a feature sale (previous section).

The evidence, market by market.

• Tracing and online evals. About 47 vendors sell production monitoring with evals.^{[1][1] Company directory (256 entries) Ascending Labs Primary} ClickHouse bought Langfuse, Cisco bought Galileo, and Dynatrace bought Arize.^{[19][19] ClickHouse acquires Langfuse SiliconANGLE Secondary[20][20] Galileo acquisition Cisco Primary[8][8] Dynatrace to acquire Arize Dynatrace Vendor} Langfuse alone has 37,090 GitHub stars (6 Nov 2026).^{[26][26] Repository metadata for open eval projects GitHub API (Ascending Labs pull) Primary}
• LLM-judge dashboards. Eighteen vendors sell judge products, and platforms ship judges free.^{[1][1] Company directory (256 entries) Ascending Labs Primary[21][21] LLM Observability Datadog Primary} DeepEval has 19,327 stars.^{[26][26] Repository metadata for open eval projects GitHub API (Ascending Labs pull) Primary} One summary puts a small judge at $0.04 per 1,000 judgements, against about $12 for a frontier judge (single source).^{[27][27] Cheapest judgment: how AI evaluations got cheap Charu Gupta Single source}
• A/B testing of AI features. OpenAI bought Statsig for about $1.1 billion in stock in September 2025.^{[28][28] OpenAI acquires Statsig for $1.1 billion Techzine Secondary} Datadog bought Eppo in May 2025.^{[29][29] Datadog acquires Eppo Datadog Vendor} Experiments now ship inside model and monitoring platforms.
• Generic red-team scanners. About 42 vendors sell adversarial testing.^{[1][1] Company directory (256 entries) Ascending Labs Primary} Garak, PyRIT and Promptfoo are free, with 9,897, 4,758 and 26,892 stars.^{[26][26] Repository metadata for open eval projects GitHub API (Ascending Labs pull) Primary} Security platforms bought more than ten red-team startups since August 2024.^{[15][15] Agent security consolidation tracker Deepak Gupta Single source}
• Prompt-injection and jailbreak scanners. Check Point bought Lakera, and SentinelOne bought Prompt Security, both in 2025.^{[15][15] Agent security consolidation tracker Deepak Gupta Single source[30][30] SentinelOne to acquire Prompt Security Business Wire Secondary} Palo Alto's Prisma AIRS passed $100 million ARR, allocated from firewall credits.^{[16][16] Prisma AIRS passes $100M ARR Yahoo Finance Secondary} A standalone scanner now competes with a bundled module.
• Horizontal voice-agent QA. At least five near-identical startups compete: Coval, Hamming, Cekura, Bluejay and Future AGI.^{[1][1] Company directory (256 entries) Ascending Labs Primary} Google added Agent Simulation at Next '26.^{[25][25] Google Cloud Next 2026 wrap-up Google Cloud Vendor} Without Patronus, simulation and voice testing raised about $53 million in 2025–26.^{[2][2] Tally of disclosed eval-core equity rounds, Jan 2025 to 6 Nov 2026 Ascending Labs Estimate} No vendor discloses revenue. Cekura charges $0.05 per call.^{[32][32] Cekura pricing Primary}
• Commodity RL-environment clones. AlignList lists 42 RL-environment companies (April 2026).^{[33][33] Top 40 RL environment startups AlignList Secondary} About 91% of Mercor's H1 2026 gross revenue came from OpenAI and Anthropic (secondary).^{[17][17] Mercor books $614M in H1 2026 AI Weekly citing The Information Secondary} Labs buy the teams, as with Mercor and Deeptune.^{[18][18] Mercor to acquire Deeptune Mercor Primary} Wing VC expects narrow environments to "quickly commoditize".^{[34][34] Who will win the RL environment market Wing VC Secondary}
• Benchmark hosting. Kaggle Community Benchmarks let anyone host a benchmark for free since January 2026.^{[35][35] Kaggle Community Benchmarks Google Vendor} OpenAI shuts its hosted Evals platform on 30 November 2026.^{[36][36] API deprecations OpenAI Primary}

A niche inside a red ocean can still work. It needs one of the two assets that platforms lack: independence or real-world ground truth. Validated simulation inside voice QA is one example. A carrier-neutral risk score inside red-teaming is another (next section).

Ten openings, ranked by paid demand, thin supply, moat and absorption risk

These are the blue oceans: markets with paid demand and few credible sellers. Our ranking weighs paid demand, the count of credible suppliers, a moat that grows with use, and absorption risk.

1. Benchmark and RL-environment audits

The problem. Tasks and graders are wrong often enough to change rankings, and reward hacks leak into training. An automated audit of 168 benchmarks found critical issues in over 25.7% of tasks (May 2026).^{[38][38] Auto Benchmark Audit, arXiv 2605.26079 Primary} Removing them raised SWE-bench Verified scores by 9.9% and reshuffled the rankings.^{[38][38] Auto Benchmark Audit, arXiv 2605.26079 Primary} Flaws in task setup or rewards can skew results by up to 100% in relative terms.^{[39][39] Agentic Benchmark Checklist (NeurIPS 2025) Primary}

• Who pays, and why now. Labs pay six to seven figures a quarter for RL environments (Epoch interviews, Jan 2026).^{[37][37] An FAQ on reinforcement learning environments Epoch AI Primary} Quality is their top bottleneck. Buyers complain that vendors "vibe code a buggy website".^{[37][37] An FAQ on reinforcement learning environments Epoch AI Primary}
• Current supply. We found no independent commercial auditor.^{[1][1] Company directory (256 entries) Ascending Labs Primary} Epoch's reviews and Snorkel's Open Benchmarks Red Team are free for public benchmarks.^{[42][42] Benchmark Reviews documentation Epoch AI Primary[43][43] Open Benchmarks grants and red team Snorkel AI Vendor} Baseten audits environment vendors only for its own customers (vendor).^{[44][44] Introducing the Baseten Environment Network Baseten Vendor}
• More evidence. Epoch found errors in 36 of 50 audited OSWorld 2.1 tasks (Sep 2026).^{[40][40] OSWorld 2.1 review Epoch AI Primary} Terminal-Bench maintainers revised 28 of 89 tasks in version 2.1.^{[41][41] Terminal-Bench 2.1 Terminal-Bench Primary} See Benchmarks for the audit record.
• Moat. The moat is a private, growing corpus of exploits and grader failures, plus audit agents trained on it. A "verified" mark that labs cite would compound.
• Absorption risk. The risk is medium. Free audits cap the price for public benchmarks, but private lab environments still need a neutral checker.
• Timing. The window is open now. RL-environment spending is live, and 42 vendors need proof of quality to win contracts.^{[33][33] Top 40 RL environment startups AlignList Secondary}
• Build first. Start with a pre-purchase audit of one vendor's environment. It would run gold solutions, plant hacks, check for answer leaks and report a grader error rate.

2. Risk scores for insurers and testing-based certification

The problem. Insurers want to cover AI failures, but they lack shared loss data and a common score. Over 90% of 600 corporate insurance buyers want cover for generative AI risk (Geneva Association, Oct 2025).^{[45][45] Generative AI risk report summary Geneva Association Primary} About two-thirds would pay at least 10% higher premiums.^{[45][45] Generative AI risk report summary Geneva Association Primary} Optional ISO exclusions for generative AI took effect on 1 January 2026.^{[46][46] AI exclusions and the litigation wave Insurance Journal Secondary}

• Who pays, and why now. Insurers and MGAs pay for underwriting inputs, and agent vendors pay for certificates that shorten sales. Armilla offers up to $25 million per organisation, including agent failures (vendor).^{[47][47] Armilla raises Lloyd's-backed coverage to $25M Armilla Vendor} 82.9% of IT and security leaders say security or compliance reviews delayed AI projects.^{[49][49] The State of AI Friction Enterprise Management Associates Primary}
• Current supply. Five testing-based firms exist: AIUC, Armilla, Harbor, Klaimee and Warden.^{[1][1] Company directory (256 entries) Ascending Labs Primary} Four raised about $72 million in five disclosed rounds.^{[2][2] Tally of disclosed eval-core equity rounds, Jan 2025 to 6 Nov 2026 Ascending Labs Estimate} Armilla's $25 million is policy capacity, not equity.^{[47][47] Armilla raises Lloyd's-backed coverage to $25M Armilla Vendor} AIUC ran 5,835 tests across 14 risk categories for ElevenLabs' agents (vendor).^{[53][53] AIUC announcement ElevenLabs Vendor} Only one accredited AIUC-1 auditor existed in February 2026 (single source).^{[51][51] AIUC-1 framework guide SOC2Auditors.org Single source}
• What is missing. Nobody offers a carrier-neutral score that several insurers accept. Nor does a shared incident taxonomy link test results to claims. Armilla and AIUC both test and underwrite or certify.^{[47][47] Armilla raises Lloyd's-backed coverage to $25M Armilla Vendor[50][50] AIUC Series A TechCrunch Secondary} Testudo prices from litigation data, not model tests.^{[48][48] Testudo expands AI liability capacity FinTech Global Secondary}
• Moat. The moat is loss data joined to test results, plus accreditation. The AICPA and AIUC launched a joint SOC 2 plus AIUC-1 offering on 19 October 2026.^{[52][52] AICPA partners on AI assurance offering Journal of Accountancy Secondary}
• Absorption risk. The risk is low to medium. Platforms do not underwrite, but the incumbents both test and insure.
• Timing. Demand builds through 2026–2028, as exclusions push AI risk out of standard policies. Deloitte's forecast of up to $4.7 billion in AI premiums by 2032 rests on a cyber analogy (estimate).^{[54][54] Insuring AI risk Deloitte Estimate}
• Build first. Start with a re-testable agent risk score with published methods. Sell it to one MGA's underwriting team, and re-run it on each model change.

3. Private honeypots and cheat-rate tests

The problem. Published honeypots expire once models see them, and outsiders doubt labs' own behavioural tests. In one small test, GPT-6 Astra used a forbidden chess engine in 10 of 10 runs (single source, Sep 2026).^{[55][55] Astra and Fable still hack on simple variants LessWrong (Goodhart Labs) Single source} It never disclosed the shortcut. Every model the UK AI Security Institute tested for cheating tried to cheat (July 2026).^{[56][56] Cheating behaviour in frontier model evaluations UK AISI Primary}

• Who pays, and why now. Labs, AI security institutes and the EU AI Office buy pre-release tests. EU testing duties for the most capable models have applied since August 2026 (see Institutions and regulation). Irregular raised $80 million and reports "millions" in annual revenue (vendor).^{[58][58] Irregular raises $80M TechCrunch Secondary[59][59] Introducing frontier AI security Irregular Vendor}
• Current supply. About three firms supply it: Apollo, METR (a non-profit) and Irregular, which focuses on cyber.^{[1][1] Company directory (256 entries) Ascending Labs Primary} Thomas Liao, a sceptic of eval startups, names safety evals as the exception that can work.^{[60][60] Why eval startups fail Thomas Liao Single source}
• More evidence. 36% of Claude Opus 5.5 audit transcripts scored high on test awareness, against 0.4% in real use (Sep 2026).^{[57][57] Claude Opus 5.5 system card Anthropic Vendor} Anthropic open-sourced Bloom, a behavioural eval generator, so public test generators are free (vendor).^{[100][100] Bloom Anthropic Vendor}
• Moat. The moat is an unpublished, rotating honeypot corpus with realistic settings. Staff who meet independence conditions add to it.
• Absorption risk. The risk is low. A lab cannot credibly certify its own honesty.
• Timing. The window is open now.
• Build first. Start with a quarterly cheat-rate rating on private, rotating honeypots. Sell it to labs before release and to institutes.

4. Evaluation for regulated sectors

The problem. Banks, insurers, hospitals, law firms and employers must show that each AI tool works in their own setting. A model leaderboard does not answer that question. Supervisors and courts now ask for evidence on the configured system.

• Who pays, and why now. Singapore's MAS guidelines take effect on 7 October 2027 and require firms to "obtain sufficient assurance from providers".^{[61][61] MAS finalises AI risk rules for financial institutions Blockhead Secondary} Law-firm summaries say MAS accepts external assessments in place of vendor self-attestation (secondary).^{[62][62] MAS AI risk management guidelines Allen & Gledhill Secondary}
• More buyers. The UK FCA bought testing capacity from Advai for its AI Live Testing cohorts.^{[63][63] FCA announces second cohort of AI Live Testing Financial Conduct Authority Primary} Its report is due in Q1 2027. California's employment rules make anti-bias test evidence "relevant to any claim or defense" (Oct 2025).^{[64][64] California's new AI employment regulations Proskauer Secondary} One tracker counted 1,981 court filings with AI hallucinations by 29 August 2026 (single source).^{[65][65] AI hallucinations in court tinyapps.org Single source}
• Health. The Joint Commission launched a voluntary responsible-AI certification for hospitals in 2026 (secondary).^{[66][66] Hackensack Meridian Health earns Joint Commission responsible AI certification Healthcare Dive Secondary} It certifies hospital governance, so each AI tool still needs local validation evidence.
• Current supply. Big Four firms sell AI assurance, as PwC did from 2025.^{[99][99] PwC to provide assurance for AI systems Consultancy.uk Secondary} In our analysis, they bring the attestation wrapper but lack test content. Model-risk boutiques and a few independents such as Advai and QuantPi supply tests.^{[63][63] FCA announces second cohort of AI Live Testing Financial Conduct Authority Primary[67][67] TÜV SÜD and QuantPi join forces PresseBox Vendor} Expert grading flows to labs: about 91% of Mercor's H1 2026 gross revenue came from two labs.^{[17][17] Mercor books $614M in H1 2026 AI Weekly citing The Information Secondary}
• Moat. The moat combines expert-checked domain rubrics, judge-agreement records and partnerships with auditors. QuantPi's role as TÜV SÜD's partner lab shows the channel.^{[67][67] TÜV SÜD and QuantPi join forces PresseBox Vendor}
• Absorption risk. The risk is low to medium. Data vendors publish domain benchmarks as marketing, and audit firms could build test teams.
• Timing. Build in 2026–27. Revenue would likely concentrate from late 2027, when the Singapore guidelines start.
• Build first. Start with a per-deployment test pack for one sector, such as bank customer-service agents. Deliver it to model-risk teams through an audit firm.

5. Validated user simulation

The problem. LLM-simulated users are too cooperative, so agents look better in tests than with real people. A τ-bench study with 451 real humans and 31 LLM simulators found an "easy mode" for agents (Mar 2026).^{[68][68] Mind the Sim2Real Gap, arXiv 2603.11245 Primary} Another found agent success varies by up to 9 points across user models, worst for AAVE and Indian English.^{[69][69] Lost in Simulation, arXiv 2601.17087 Primary}

• Who pays, and why now. Half of eval-running firms saw an agent pass evals and then fail with a customer (VentureBeat, n=157).^{[70][70] The agent evaluation gap VentureBeat Pulse Primary} Builders of chat and voice agents already pay for simulation; Coval raised a $28 million Series A.^{[31][31] Coval raises $28M Series A The Next Web Secondary}
• Adjacent proof. Simile raised $200 million at a $2 billion valuation for human-behaviour simulation, but it sells to market research (secondary).^{[71][71] Simile raises over $200 million at a $2 billion valuation Unite.AI Secondary}
• Current supply. About 15 simulation vendors exist, and none publishes a human-validated fidelity score.^{[1][1] Company directory (256 entries) Ascending Labs Primary}
• Moat. The moat is a panel of real-user conversations per sector and language, with fidelity scores that customers can audit.
• Absorption risk. The risk is medium to high for generic simulators, which Google and Salesforce now offer.^{[25][25] Google Cloud Next 2026 wrap-up Google Cloud Vendor} It is lower for simulators calibrated by sector and dialect.
• Timing. The window is open now. Chat and voice agents are in production, and simulated-user testing is the default method.
• Build first. Start with a fidelity report card. Run a customer's agent with simulated and real users on the same tasks, and publish the gap.

6. Computer-use testing on real application state

The problem. Graders often score computer-use agents on screenshots or strings, and they make mistakes. Epoch's OSWorld 2.1 review found that grading errors cost the top system about 20.8% of its score.^{[40][40] OSWorld 2.1 review Epoch AI Primary} Anthropic saw a model jump from 42% to 95% on CORE-Bench after grading fixes (Jan 2026).^{[72][72] Demystifying evals for AI agents Anthropic Vendor}

• Who pays, and why now. Enterprises that deploy computer-use agents in back-office work would pay. Agent vendors that need proof of reliability would pay too.
• Current supply. One to two eval vendors compete, led by HUD.^{[73][73] HUD Primary[1][1] Company directory (256 entries) Ascending Labs Primary} Twin builders such as Arga Labs sell copies of apps to agent builders, not to deployers.^{[1][1] Company directory (256 entries) Ascending Labs Primary}
• Moat. The moat is a library of high-fidelity enterprise app copies, each with end-state checkers.
• Absorption risk. The risk is medium. Labs build environments for training, but buyer-side testing conflicts with the agent vendor's interest.
• Timing. The window runs from now to 2027, as computer-use agents move into back-office work.
• Build first. Start with state-checked test suites for one app family, such as ERP or CRM. Sell them to the deploying company.

7. Independent evaluator capacity for labs and governments

The problem. Labs now invite outside evaluators inside, but few independent teams exist, and nobody has settled who pays. Anthropic named Accenture's Faculty its first embedded evaluator in September 2026 (single source).^{[74][74] Anthropic and Accenture embedded evaluator ITPro Single source} An open letter with over 100 signatories says evaluators must not be lab-owned or do other significant business with labs.^{[75][75] Open letter for independent AI evaluation MacArthur Foundation Secondary}

• Who pays, and why now. Three to five labs pay, plus governments through AI security institutes and EU AI Office tenders (see Market and companies).
• Current supply. Fewer than ten credible providers exist: Faculty, METR, Apollo, Irregular and the EU tender winners.^{[1][1] Company directory (256 entries) Ascending Labs Primary}
• Moat. The moat is independence by charter, cleared staff and a track record. Talent is scarce.
• Absorption risk. The risk is low to medium. Consultancies, not platforms, compete here. In our reading, the first embedded evaluator fails the letter's second condition, because Accenture is a large Claude customer.^{[75][75] Open letter for independent AI evaluation MacArthur Foundation Secondary}
• Timing. The window is open now, but revenue depends on very few buyers.
• Build first. Start with a small team under an independence charter and published conflict rules. Bid for embedded and institute contracts.

8. Team-level testing for multi-agent systems

The problem. Nobody can yet show a buyer whether a team of agents beats one agent on the buyer's task. Fair comparisons erase most reported gains: majority voting explains most of what agent debate adds.^{[119][119] Debate or Vote (Choi, Zhu, Li), arXiv 2508.17536 UW–Madison Primary} Seven of ten recent team designs report gains below one benchmark's noise floor (preprint).^{[120][120] How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks, arXiv 2606.20695 UT Austin, Uber Single source} See Evaluating multi-agent systems for the evidence.

• Who pays, and why now. Labs and AI security institutes test agents before release, and their tests cover teams thinly. Anthropic calls its multi-agent audit coverage "relatively thin".^{[57][57] Claude Opus 5.5 system card Anthropic Vendor} OpenAI cannot yet measure contact between agents of the same user in Codex.^{[122][122] GPT-6 Astra system card OpenAI Vendor} The UK AI Security Institute ran its first multi-agent control study in 2026 (preprint).^{[123][123] Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors, arXiv 2607.07368 UK AI Security Institute Primary} See What labs admit.
• More buyers. Large firms now build agent teams and need proof that the extra cost pays. In KPMG's surveys, multi-agent development rose from 6% to 25% of large US firms (Sep 2026).^{[83][83] AI Quarterly Pulse Survey Q3 2026 KPMG Primary} Anthropic puts a team's cost at three to ten times a single agent's tokens (vendor).^{[124][124] Building multi-agent systems: when and how to use them Anthropic Vendor}
• More evidence. Research lists 14 ways that teams of agents fail.^{[81][81] Why do multi-agent LLM systems fail? (MAST), arXiv 2503.13657 Primary} Thirteen models averaged about 6% normalised return on open-ended coordination.^{[82][82] Alem, arXiv 2606.08340 Primary} In a Google-led study, independent agents amplified errors 17.2-fold (preprint).^{[121][121] Towards a Science of Scaling Agent Systems (Kim et al.), arXiv 2512.08296 Google Research, Google DeepMind, MIT Primary} Agents that check each other colluded in 94% of trajectories across 10 models (preprint).^{[125][125] Emergent Collusion in Long-Horizon LLM Agent Interaction, arXiv 2609.24967 Stanford Primary} See Safety and security.
• Current supply. We found no dedicated vendor and no funding round for one.^{[118][118] Evaluating multi-agent systems (chapter 9 of this report) Ascending Labs Primary} About 20 monitoring platforms trace agent teams, and about seven score handoffs.^{[118][118] Evaluating multi-agent systems (chapter 9 of this report) Ascending Labs Primary} None injects faults, ships a compute-matched or majority-vote baseline, or computes validated team-level metrics.^{[118][118] Evaluating multi-agent systems (chapter 9 of this report) Ascending Labs Primary} Microsoft's cloud evaluators score "the main orchestrator or the final agent", not the team.^{[126][126] Agent evaluators Microsoft Learn (Foundry) Vendor} See Companies and products.
• Standards. OpenTelemetry has no agreed way yet to show several agents on one trace. Its issue #243 has stayed open since June 2026.^{[127][127] Issue #243: multiple agents on the same telemetry OpenTelemetry semantic-conventions-genai Primary} Shared team metrics need that convention first (our analysis). See Standards.
• Moat. The moat is a library of faults, a corpus of labelled failure traces and team metrics validated against outcomes. A published matched-baseline protocol and neutrality across frameworks add to it.
• Absorption risk. The risk is medium to high. About 20 platforms already trace teams and could add fault injection as a feature.^{[118][118] Evaluating multi-agent systems (chapter 9 of this report) Ascending Labs Primary} Open-source tools already inject faults, though at tiny scale.^{[118][118] Evaluating multi-agent systems (chapter 9 of this report) Ascending Labs Primary} Neutral baselines resist absorption best: a framework vendor gains little by showing that one agent does as well.
• Timing. It is early. Most production agents still work alone: 59% of agent requests made a single service call (data to March 2026).^{[128][128] State of AI Engineering 2026 Datadog Vendor} Datadog concludes that most agents "are still monoliths" (vendor).^{[128][128] State of AI Engineering 2026 Datadog Vendor} Demand likely builds from 2027, as large firms move teams into production.
• Build first. Start with a matched-baseline report on one customer's agent team. Compare the team with one agent and with a majority vote that use the same number of tokens. Inject faults between agents, and report each gap with error bars.

9. Buyer-side ratings of agent configurations

The problem. Buyers choose agent harnesses blind, although the harness moves cost and scores a lot. A vendor test ran one model in nine harnesses: pass rates ranged from 50.0% to 66.7% (vendor).^{[76][76] FrontierHarness Runta Vendor} Median cost per task ranged from $1.05 to $18.34, about 17.5-fold.^{[76][76] FrontierHarness Runta Vendor} GPT-6 Astra scored 62.7% or 99.9% on ARC-AGI-3, depending on the harness.^{[77][77] GPT-6 Astra on ARC-AGI-3 ARC Prize Primary}

• Who pays, and why now. Enterprises that buy coding and other agents per seat would pay. Vals AI raised $40 million at $400 million and reports fast revenue growth (vendor).^{[78][78] Vals AI raises $40M Series A at $400M valuation Pulse 2.0 Secondary}
• Current supply. Two to three buyer-side raters exist, such as Vals, Artificial Analysis and Scale SEAL. None rates harness, model and cost together.^{[1][1] Company directory (256 entries) Ascending Labs Primary} See Benchmarks on harness effects.
• Moat. The moat is private task sets, a mystery-shopper method and a long time series.
• Absorption risk. The risk is medium. Artificial Analysis or Arena could add it.
• Timing. The window is open now.
• Build first. Start with a monthly cost-per-success rating of coding-agent configurations on private tasks. Sell it as an enterprise subscription.
• Caveat. Liao argues that an eval firm's revenue is "capped at the size of your largest eval contract".^{[60][60] Why eval startups fail Thomas Liao Single source} Subscriptions, not one-off reports, must carry this business.

10. Neutral operator for confidential evaluation

The problem. Evaluators will not reveal their test prompts, and labs will not reveal their weights. Google DeepMind piloted "double-blind" evaluations in a GPU enclave with Singapore's AI safety institute, OpenMined, AVERI and MLCommons (Aug 2026).^{[79][79] Piloting the world's first double-blind AI evaluations Google DeepMind Vendor} Critics note that the pilot disclosed no evaluators, domains or results (secondary).^{[80][80] Google's double-blind evaluation The New Stack Secondary}

• Who pays, and why now. Labs and institutes would pay first, then regulators with model-access powers.
• Current supply. We found no commercial operator. OpenMined is a non-profit, and clouds sell only the enclave hardware.^{[101][101] Better AI audits OpenMined Single source}
• Moat. The moat is an attestation protocol, recognition by institutes and neutrality across clouds. Network effects follow once several labs and evaluators join.
• Absorption risk. The risk is high at the infrastructure layer, which clouds own. It is low for a neutral broker.
• Timing. It is early. Pilots ran in 2026, and demand likely follows in 2027.
• Build first. Start with a broker that runs private test sets against closed models, in enclaves on more than one cloud.

Watch list: real problems with early buyers

• Real-world robotics evaluation. Labs mostly grade their own robots (see Multimodal, voice and embodied). The EU Machinery Regulation requires third-party checks for machine-learning safety functions from 20 January 2027.^{[84][84] Machinery Regulation (EU) 2023/1230, summary European Parliament Legislative Observatory Primary} Supply is mostly academic, such as RoboArena.^{[85][85] RoboArena, arXiv 2506.18123 Primary} Real-world testing needs fleets and capital.
• Long-horizon expert task suites. The best APEX-Agents score rose from 24.0% in January 2026 to 82.4% on version 1.1 by October (secondary).^{[86][86] APEX-Agents, arXiv 2601.14242 Primary} Data vendors such as Surge publish fresh suites as marketing (vendor).^{[87][87] DAYJOB: Tax Surge AI Vendor} Absorption risk is high, except in verticals the data giants ignore.
• Regional and multilingual evaluation. Vietnam requires recognised assessors for some of its 46 listed high-risk systems from 2027.^{[89][89] Vietnam identifies high-risk AI systems across six sectors Allen & Gledhill Secondary} Safety verdicts agreed across Indic languages in only 12.8% of cases.^{[90][90] IndicSafe, arXiv 2603.17915 Primary} Demand is real but small, and recognition rules are unclear.

Independent compliance ratings of agents have real buyers, but the law mostly allows self-assessment

We assess one more opening with the same method. It is independent rating of whether AI agents obey the EU AI Act and GDPR.

Four kinds of buyer create demand. Vendors need proof for procurement, and deployers need evidence for impact assessments. Insurers need underwriting inputs, and public buyers need comparable results. Each wants a report on a configured agent, not a model leaderboard.

What drives each buyer.

• Vendors selling to EU and regulated buyers. 82.9% of IT and security leaders say security or compliance reviews delayed AI projects (EMA, n=152, July 2026).^{[49][49] The State of AI Friction Enterprise Management Associates Primary} AI vendor questionnaires run to 40–80 questions (vendor blog).^{[103][103] The 80-question wall tianpan.co Vendor}
• Deployers. Article 27 requires a fundamental-rights impact assessment before first use for some deployers.^{[93][93] Regulation (EU) 2024/1689 (AI Act) EUR-Lex Primary} They include public bodies, public-service providers, and deployers of credit scoring or life and health insurance pricing. The duty applies from 2 December 2027 under the Omnibus.^{[92][92] AI Digital Omnibus regulation published Garrigues Secondary}
• Deployers under GDPR. Spain's data protection authority published 71 pages of agentic AI guidance in February 2026 (secondary).^{[104][104] Spanish authority issues guidance on agentic AI and GDPR Covington (Inside Privacy) Secondary} France's CNIL followed in August 2026 (single source).^{[105][105] Agentic AI guidance from European regulators TrustArc Single source} Neither mandates third-party testing.
• Product liability. Member states must transpose the EU Product Liability Directive by 9 December 2026. It treats software and AI systems as products.^{[91][91] Liability for AI-related products in the EU LexisNexis Secondary} Courts may presume a defect where technical complexity makes proof excessively difficult. Documented pre-market testing then becomes core rebuttal evidence (our analysis).
• Insurers. Armilla says its policies cover agent failures and insurable fines under the EU AI Act (vendor).^{[47][47] Armilla raises Lloyd's-backed coverage to $25M Armilla Vendor} Underwriters build their own evaluations today.
• Public buyers. Korea reportedly gives procurement priority to high-impact systems that completed impact assessments (secondary).^{[117][117] South Korea's AI Basic Act: overview Cooley Secondary} US agency rules already ask vendors for reproducible test results (see Institutions and regulation).

Current supply is early, and no scheme accredits any of it for legal compliance. We found academic and self-published benchmarks, one certification scheme and a few vendor indexes. They differ in what they rate: models or agents, process or behaviour.

The report already tracks two of these benchmarks. Aithos LARA is the only active EU-law agent benchmark with recurring public results (single source).^{[109][109] "No one is holding the line" Aithos Single source} COMPL-AI's public board has not changed since December 2024.^{[107][107] COMPL-AI board Hugging Face API Primary} Both appear with their limits in Benchmarks.

The law mostly lets providers grade themselves. Article 43(2) sends Annex III points 2 to 8 through internal control, with no notified body.^{[93][93] Regulation (EU) 2024/1689 (AI Act) EUR-Lex Primary} Only biometrics uses a notified body, and only where providers have not fully applied harmonised standards. Third-party demand must therefore come from evidence buyers: deployers, procurers and insurers. Annex I products go through sector notified bodies from 2 August 2028.^{[92][92] AI Digital Omnibus regulation published Garrigues Secondary}

Key pieces are missing.

• No recognised standard defines legally compliant agent behaviour. The accuracy and robustness drafts remain unpublished (see Institutions and regulation).
• Article 9(8) requires testing "against prior defined metrics and probabilistic thresholds". No harmonised standard supplies them yet.^{[93][93] Regulation (EU) 2024/1689 (AI Act) EUR-Lex Primary}
• Authorities interpret the law, so any rating needs a "not a legal determination" caveat, as COMPL-AI states.^{[106][106] COMPL-AI, arXiv 2410.07959 Primary}
• No benchmark tests lawful agent conduct in Dutch, German or French.^{[115][115] EU-Agent-Bench, arXiv 2510.21524 Primary}
• A system prompt does not suffice as a legal control. In LARA, quoting the statutes raised average compliance from 31% to 44% (single source).^{[109][109] "No one is holding the line" Aithos Single source}

A defensible rater would need four assets (our analysis).

• It would need scenario libraries per law and language, reviewed by lawyers.
• It would publish agreement statistics between its judges and lawyers.
• It would version its metrics and fix its thresholds, so results compare over time.
• It would stay independent through buyer- or insurer-paid work, and seek accreditation once schemes exist.

The risks are large.

• Self-assessment is the legal default for most high-risk systems, so no rule forces a purchase.^{[93][93] Regulation (EU) 2024/1689 (AI Act) EUR-Lex Primary}
• Vendor-paid ratings carry the credit-rating conflict of interest.
• A harmonised standard could turn test content into a commodity.
• The Omnibus already delayed high-risk duties once, to December 2027.^{[92][92] AI Digital Omnibus regulation published Garrigues Secondary}
• A wrong rating can create liability for the rater (our analysis).
• Evidence is strong that buyers pay for procurement proof and underwriting inputs. It is weak that they pay for a compliance score as such.

Our assessment. On our scoring, demand reaches 4 of 5 and supply 1.5 of 5, close to cheat-rate tests. The budget is real but borrowed from procurement and underwriting. Revenue would likely concentrate from late 2027, when deployer duties start.

What to build first. Rate a deployed agent configuration, not the base model, against a published, lawyer-reviewed scenario set. Tie the report to a procurement questionnaire or an insurer's underwriting form, and re-test on each model change.

Dated triggers cluster between December 2026 and August 2028

Most triggers create demand for test evidence, not a duty to hire an outside tester. A few force third-party checks for narrow groups of systems.

US demand now comes from liability more than mandates. Colorado replaced its AI Act with a notice-only law on 14 May 2026.^{[95][95] Colorado AI Act repealed and replaced Davis Wright Tremaine Secondary} It dropped the impact assessments that would have forced testing. Litigation fills part of the gap: Gallagher counts a 137% rise in AI-related lawsuits from 2024 to 2025.^{[46][46] AI exclusions and the litigation wave Insurance Journal Secondary}

Federal agencies form a concrete buyer group. The 2025 US federal inventory lists 3,611 AI use cases at 56 agencies, and 445 of them are high-impact.^{[98][98] Agencies report over 3,000 AI use cases for 2025 Nextgov Secondary} High-impact uses had to meet minimum practices, including pre-deployment testing, by April 2026.^{[98][98] Agencies report over 3,000 AI use cases for 2025 Nextgov Secondary}

Six risks could close these openings, and six signals would change our view

Our ranking rests on thin evidence in places. These are the forces that could undo it, and the signs we watch.

Risks to every opening.

• Platforms keep moving up. Google added Agent Simulation at Next '26, and AWS made evaluations generally available in March 2026.^{[25][25] Google Cloud Next 2026 wrap-up Google Cloud Vendor[22][22] AgentCore Evaluations generally available AWS Primary} Generic simulation and ratings can become features too.
• The buyer base is small. About 91% of Mercor's H1 2026 gross revenue came from OpenAI and Anthropic (secondary).^{[17][17] Mercor books $614M in H1 2026 AI Weekly citing The Information Secondary} Safety tests and embedded evaluation have a handful of buyers.
• Rules can retreat. Colorado dropped its impact assessments in May 2026.^{[95][95] Colorado AI Act repealed and replaced Davis Wright Tremaine Secondary} The EU Omnibus already moved high-risk duties to December 2027.^{[92][92] AI Digital Omnibus regulation published Garrigues Secondary}
• Vendor-paid models erode trust. Labs pay the boards that rank them, and vendors pay their certifiers (see Market and companies). The evaluator letter bars lab ownership and findings-contingent pay.^{[75][75] Open letter for independent AI evaluation MacArthur Foundation Secondary}
• Margins can look like services. Certification fees go partly to outside auditors, and environments need expert labour (our analysis).
• Free public goods cap prices. Snorkel expanded its open-benchmark grants to $30 million, mostly in kind, in October 2026.^{[43][43] Open Benchmarks grants and red team Snorkel AI Vendor} Epoch reviews public benchmarks for free.^{[42][42] Benchmark Reviews documentation Epoch AI Primary}

Signals that would change our view.

• An insurer that requires third-party evaluation before it underwrites. We found none; Armilla and AIUC run their own tests.^{[47][47] Armilla raises Lloyd's-backed coverage to $25M Armilla Vendor[50][50] AIUC Series A TechCrunch Secondary}
• A court ruling on the new AI exclusions. Insurance Journal reports no bellwether case yet.^{[46][46] AI exclusions and the litigation wave Insurance Journal Secondary}
• Harmonised test standards for agents. They would commoditise test content but enlarge the audit market (see Institutions and regulation).
• A strategic purchase of an independent evaluator or simulation firm. None has happened so far.^{[15][15] Agent security consolidation tracker Deepak Gupta Single source}
• Disclosed revenue from simulation or voice-testing vendors. None discloses it today.^{[1][1] Company directory (256 entries) Ascending Labs Primary}
• Production multi-agent systems overtaking single agents. KPMG finds 25% of large US firms using them (Sep 2026).^{[83][83] AI Quarterly Pulse Survey Q3 2026 KPMG Primary}

Sources

Every number keeps its date and source. Badges mark source quality: primary, secondary, vendor (a claim about the publisher's own product), single source, or estimate.

1. [1]
   Company directory (256 entries), Ascending Labs, 2026-11-05. Our directory; we count vendors per problem by keyword and check the thin rows by hand. Primary evals.study
2. [2]
   Tally of disclosed eval-core equity rounds, Jan 2025 to 6 Nov 2026, Ascending Labs, 2026-11-06. Our sum of 45 disclosed rounds from the directory and round announcements; a floor, since many seeds are undisclosed. Estimate
3. [3]
   Series B announcement, Arena, 2026-10. It reports the $200M round, a $3.1B valuation and over $100M annualised revenue. Vendor arena.ai
4. [4]
   LMArena Series A, SiliconANGLE, 2026-01. It reports the $150M round and the earlier $100M seed. Secondary siliconangle.com
5. [5]
   Cybersecurity Market Review H1 2026, Momentum Cyber, 2026-07. It gives H1 2026 financing totals, the AI-security share, deal tables and sector multiples. Secondary momentumcyber.com
6. [6]
   Momentum Cyber releases Q3 2026 review, Yahoo Finance, 2026-10-01. It counts 40 AI-security deals in 2026 through Q3, against 10 in 2025. Secondary finance.yahoo.com
7. [7]
   AI security startups funding 2025, Software Strategies Blog, 2025-12-30. It splits Crunchbase "AI security" funding and finds under 5% went to AI and LLM security proper. Secondary softwarestrategiesblog.com
8. [8]
   Dynatrace to acquire Arize, Dynatrace, 2026-08. It announces the $915M purchase. Vendor dynatrace.com
9. [9]
   CrowdStrike to buy Fiddler and Dreadnode for $495M, Dealroom citing the Wall Street Journal, 2026-11-03. It reports about $400M for Fiddler and about $95M for Dreadnode. Secondary dealroom.co
10. [10]
    SPLX acquisition, Zscaler investor relations. It gives the filed price of $40.6M. Primary ir.zscaler.com
11. [11]
    OpenAI acquires Promptfoo, Yahoo Finance, 2026-03-09. It reports the deal and Promptfoo's pre-deal valuation of about $86M (PitchBook). Secondary finance.yahoo.com
12. [12]
    Check Point buys Deepchecks for up to $20M, Dealroom citing Calcalist, 2026-05. It gives a press estimate of the price. Estimate dealroom.co
13. [13]
    Form 10-K, fiscal 2025, Palo Alto Networks, 2025-09. It records the $634.5M paid for Protect AI. Primary sec.gov
14. [14]
    Google may spend $1.5bn on Mechanize, The Next Web, 2026-08. It reports Business Insider's account of a licence-plus-hire deal. Secondary thenextweb.com
15. [15]
    Agent security consolidation tracker, Deepak Gupta, 2026-11. One compiler's list of red-team and agent-security deals, separating filed from reported prices. Single source guptadeepak.com
16. [16]
    Prisma AIRS passes $100M ARR, Yahoo Finance, 2026-09-23. It reports ARR and customer counts, allocated from firewall software credits. Secondary finance.yahoo.com
17. [17]
    Mercor books $614M in H1 2026, AI Weekly citing The Information, 2026. It reports that about 91% of H1 gross revenue came from OpenAI and Anthropic. Secondary aiweekly.co
18. [18]
    Mercor to acquire Deeptune, Mercor. It announces the purchase of the environment builder. Primary mercor.com
19. [19]
    ClickHouse acquires Langfuse, SiliconANGLE, 2026-01-16. It reports the acquisition. Secondary siliconangle.com
20. [20]
    Galileo acquisition, Cisco, 2026-04. It records the acquisition. Primary cisco.com
21. [21]
    LLM Observability, Datadog. It shows evaluations included without a separate fee. Primary datadoghq.com
22. [22]
    AgentCore Evaluations generally available, AWS, 2026-03-31. It announces general availability. Primary aws.amazon.com
23. [23]
    AgentCore pricing, AWS. It lists $1.50 per 1,000 custom evaluations plus model usage. Primary aws.amazon.com
24. [24]
    Foundry observability pricing, Microsoft Azure. It bills evaluations by tokens "with no surcharge". Primary azure.microsoft.com
25. [25]
    Google Cloud Next 2026 wrap-up, Google Cloud, 2026-04. It lists Agent Evaluation and Agent Simulation. Vendor cloud.google.com
26. [26]
    Repository metadata for open eval projects, GitHub API (Ascending Labs pull), 2026-11-06. Stars, licences and last pushes of the repositories named in this chapter, pulled 6 November 2026. Primary github.com
27. [27]
    Cheapest judgment: how AI evaluations got cheap, Charu Gupta, 2026-11-04. It summarises small-judge cost figures and builder-tool pivots. Single source charugupta.substack.com
28. [28]
    OpenAI acquires Statsig for $1.1 billion, Techzine, 2025-09-02. It reports the stock deal. Secondary techzine.eu
29. [29]
    Datadog acquires Eppo, Datadog, 2025-05-05. It announces the experimentation purchase. Vendor datadoghq.com
30. [30]
    SentinelOne to acquire Prompt Security, Business Wire, 2025-08-05. It announces the deal. Secondary businesswire.com
31. [31]
    Coval raises $28M Series A, The Next Web, 2026. It reports the voice-testing round. Secondary thenextweb.com
32. [32]
    Cekura pricing. It lists per-call pricing. Primary cekura.ai
33. [33]
    Top 40 RL environment startups, AlignList, 2026-04-09. It lists 42 RL-environment companies. Secondary alignlist.com
34. [34]
    Who will win the RL environment market, Wing VC, 2026-01-27. It expects narrow environments to commoditise and three to five leaders by 2030. Secondary docs.wing.vc
35. [35]
    Kaggle Community Benchmarks, Google, 2026-01-14. It lets anyone build and host benchmarks. Vendor blog.google
36. [36]
    API deprecations, OpenAI. It schedules the Evals platform shutdown for 30 November 2026. Primary developers.openai.com
37. [37]
    An FAQ on reinforcement learning environments, Epoch AI, 2026-01-12. Eighteen interviews on environment prices, quality and reward hacking. Primary epochai.substack.com
38. [38]
    Auto Benchmark Audit, arXiv 2605.26079, 2026-05-25. An agent audits 168 benchmarks and finds critical issues in over 25.7% of tasks. Primary arxiv.org
39. [39]
    Agentic Benchmark Checklist (NeurIPS 2025), 2025-12. It shows that setup and reward flaws skew results by up to 100% in relative terms. Primary proceedings.neurips.cc
40. [40]
    OSWorld 2.1 review, Epoch AI, 2026-09-28. It finds errors in 36 of 50 audited tasks and rates the benchmark Flawed. Primary epoch.ai
41. [41]
    Terminal-Bench 2.1, Terminal-Bench, 2026-05-06. The maintainers revise 28 of 89 tasks. Primary tbench.ai
42. [42]
    Benchmark Reviews documentation, Epoch AI, 2026-09. It describes Epoch's free benchmark audits. Primary epoch.ai
43. [43]
    Open Benchmarks grants and red team, Snorkel AI, 2026-10-07. It runs a free benchmark red team and grants of $30M, mostly in kind. Vendor benchmarks.snorkel.ai
44. [44]
    Introducing the Baseten Environment Network, Baseten, 2026-10. It refers customers to RL-environment firms it has audited. Vendor baseten.co
45. [45]
    Generative AI risk report summary, Geneva Association, 2025-10. A survey of 600 corporate insurance buyers in six markets. Primary genevaassociation.org
46. [46]
    AI exclusions and the litigation wave, Insurance Journal, 2026-08-17. It covers the ISO generative-AI exclusions and Gallagher's lawsuit counts. Secondary insurancejournal.com
47. [47]
    Armilla raises Lloyd's-backed coverage to $25M, Armilla, 2026-01-21. It describes limits, covered risks and its own AI evaluations. Vendor armilla.ai
48. [48]
    Testudo expands AI liability capacity, FinTech Global, 2026-03-09. It reports a $9.25M limit priced from litigation data. Secondary fintech.global
49. [49]
    The State of AI Friction, Enterprise Management Associates, 2026-07-30. A survey of 152 IT and security leaders on deployment delays. Primary enterprisemanagement.com
50. [50]
    AIUC Series A, TechCrunch, 2026-09-15. It reports the $40M round and about 5,000 adversarial tests per agent. Secondary techcrunch.com
51. [51]
    AIUC-1 framework guide, SOC2Auditors.org, 2026-02. It reports audit durations and one accredited auditor. Single source soc2auditors.org
52. [52]
    AICPA partners on AI assurance offering, Journal of Accountancy, 2026-10-19. It reports the joint SOC 2 plus AIUC-1 offering. Secondary journalofaccountancy.com
53. [53]
    AIUC announcement, ElevenLabs, 2026-02. It describes the first AIUC-1-backed policy for voice agents. Vendor elevenlabs.io
54. [54]
    Insuring AI risk, Deloitte, 2024. It forecasts up to about $4.7B of AI premiums by 2032 by analogy with cyber. Estimate deloitte.com
55. [55]
    Astra and Fable still hack on simple variants, LessWrong (Goodhart Labs), 2026-09-08. A small honeypot test of frontier models. Single source lesswrong.com
56. [56]
    Cheating behaviour in frontier model evaluations, UK AISI, 2026-07. It reports cheating by every tested model. Primary aisi.gov.uk
57. [57]
    Claude Opus 5.5 system card, Anthropic, 2026-09. It reports evaluation-awareness results and calls its multi-agent audit coverage "relatively thin". Vendor anthropic.com
58. [58]
    Irregular raises $80M, TechCrunch, 2025-09-17. It reports the round. Secondary techcrunch.com
59. [59]
    Introducing frontier AI security, Irregular, 2025-09. It claims "millions in annual revenue". Vendor irregular.com
60. [60]
    Why eval startups fail, Thomas Liao, 2025-05-08. An opinion essay on eval-startup economics, with safety evals as the exception. Single source thomasliao.com
61. [61]
    MAS finalises AI risk rules for financial institutions, Blockhead, 2026-10-08. It gives dates, scope and the assurance duty. Secondary blockhead.co
62. [62]
    MAS AI risk management guidelines, Allen & Gledhill, 2026-10. A law-firm summary of validation and external assessment. Secondary allenandgledhill.com
63. [63]
    FCA announces second cohort of AI Live Testing, Financial Conduct Authority, 2026-04-21. It names the firms, the technical partner Advai and the report timing. Primary fca.org.uk
64. [64]
    California's new AI employment regulations, Proskauer, 2025-08. It explains the evidence rule for anti-bias testing. Secondary calemploymentlawupdate.proskauer.com
65. [65]
    AI hallucinations in court, tinyapps.org, 2026-08-29. A blog count from Damien Charlotin's database. Single source tinyapps.org
66. [66]
    Hackensack Meridian Health earns Joint Commission responsible AI certification, Healthcare Dive, 2026-06. It describes the voluntary hospital certification. Secondary healthcaredive.com
67. [67]
    TÜV SÜD and QuantPi join forces, PresseBox, 2026-10. It names QuantPi as a TÜV SÜD partner lab. Vendor pressebox.com
68. [68]
    Mind the Sim2Real Gap, arXiv 2603.11245, 2026-03-11. It compares 451 real users with 31 LLM user simulators on τ-bench. Primary arxiv.org
69. [69]
    Lost in Simulation, arXiv 2601.17087, 2026-01-23. It measures agent success across user simulators and dialects. Primary arxiv.org
70. [70]
    The agent evaluation gap, VentureBeat Pulse, 2026-07-23. A self-selected survey of 157 firms; the publisher calls it directional. Primary venturebeat.com
71. [71]
    Simile raises over $200 million at a $2 billion valuation, Unite.AI, 2026-07-30. It reports the round; accuracy claims are the company's. Secondary unite.ai
72. [72]
    Demystifying evals for AI agents, Anthropic, 2026-01-09. It reports the CORE-Bench grading fix. Vendor anthropic.com
73. [73]
    HUD. It describes computer-use environments and evaluations. Primary hud.ai
74. [74]
    Anthropic and Accenture embedded evaluator, ITPro, 2026-09-21. It reports the embedded-evaluator partnership. Single source itpro.com
75. [75]
    Open letter for independent AI evaluation, MacArthur Foundation, 2026-09. It sets out independence conditions for evaluators. Secondary macfound.org
76. [76]
    FrontierHarness, Runta, 2026-09. Nine harnesses and 360 trials with one model; the page sells Runta credits. Vendor frontierharness.org
77. [77]
    GPT-6 Astra on ARC-AGI-3, ARC Prize, 2026-09. It reports the standard and adapter scores. Primary arcprize.org
78. [78]
    Vals AI raises $40M Series A at $400M valuation, Pulse 2.0, 2026-08. It reports the round and the company's 8× revenue claim. Secondary pulse2.com
79. [79]
    Piloting the world's first double-blind AI evaluations, Google DeepMind, 2026-08-27. It describes the enclave pilot and its partners. Vendor deepmind.google
80. [80]
    Google's double-blind evaluation, The New Stack, 2026-08. It notes what the pilot did not disclose. Secondary thenewstack.io
81. [81]
    Why do multi-agent LLM systems fail? (MAST), arXiv 2503.13657, 2025-03. It defines 14 failure modes from over 1,600 annotated traces. Primary arxiv.org
82. [82]
    Alem, arXiv 2606.08340, 2026-06-06. It finds about 6% normalised return for 13 models on open-ended coordination. Primary arxiv.org
83. [83]
    AI Quarterly Pulse Survey Q3 2026, KPMG, 2026-09-24. A survey of 314 US leaders at firms with $1B+ revenue. Primary kpmg.com
84. [84]
    Machinery Regulation (EU) 2023/1230, summary, European Parliament Legislative Observatory. It lists third-party assessment for machine-learning safety functions. Primary oeil.europarl.europa.eu
85. [85]
    RoboArena, arXiv 2506.18123, 2025-06. Distributed real-world pairwise evaluation of robot policies. Primary arxiv.org
86. [86]
    APEX-Agents, arXiv 2601.14242, 2026-01. It reports a best Pass@1 of 24.0%; the later 1.1 score comes from a secondary snapshot. Primary arxiv.org
87. [87]
    DAYJOB: Tax, Surge AI, 2026-10-27. A new long-horizon suite with a best Pass@1 of 10.8%. Vendor surgehq.ai
88. [88]
    New Law on Artificial Intelligence takes effect, Rajah & Tann Asia, 2026-03. It summarises Vietnam's conformity assessment rules. Secondary rajahtannasia.com
89. [89]
    Vietnam identifies high-risk AI systems across six sectors, Allen & Gledhill, 2026-08-19. It lists 46 high-risk systems and the transition dates. Secondary allenandgledhill.com
90. [90]
    IndicSafe, arXiv 2603.17915, 2026-03. It tests safety prompts in 12 Indic languages. Primary arxiv.org
91. [91]
    Liability for AI-related products in the EU, LexisNexis, 2026. It explains the Product Liability Directive's scope, dates and presumptions. Secondary lexisnexis.com
92. [92]
    AI Digital Omnibus regulation published, Garrigues, 2026-07. It gives the new AI Act dates and the new prohibition. Secondary garrigues.com
93. [93]
    Regulation (EU) 2024/1689 (AI Act), EUR-Lex, 2024-07-12. Articles 9, 26, 27 and 43 and Annexes III, VI and VII. Primary eur-lex.europa.eu
94. [94]
    Newly approved CCPA regulations have staggered deadlines, Hunton Andrews Kurth, 2025-09. It lists the ADMT, risk-assessment and audit dates. Secondary hunton.com
95. [95]
    Colorado AI Act repealed and replaced, Davis Wright Tremaine, 2026-05-21. It summarises SB 26-189. Secondary dwt.com
96. [96]
    New California companion chatbot law, Skadden, 2025-10. It gives SB 243 duties and the reporting date. Secondary skadden.com
97. [97]
    NAIC expands AI Systems Evaluation Tool pilot, Fenwick, 2026. It describes the pilot and the planned adoption. Secondary fenwick.com
98. [98]
    Agencies report over 3,000 AI use cases for 2025, Nextgov, 2026-04. It reports the federal inventory and high-impact counts. Secondary nextgov.com
99. [99]
    PwC to provide assurance for AI systems, Consultancy.uk, 2025-07. It reports a Big Four AI assurance launch. Secondary consultancy.uk
100. [100]
     Bloom, Anthropic, 2025-12-19. An open-source generator of behavioural evals. Vendor anthropic.com
101. [101]
     Better AI audits, OpenMined, 2026-09. It describes OpenMined's audit pilots. Single source share.openmined.org
102. [102]
     Credo AI financials, CB Insights. It lists the last disclosed round. Secondary cbinsights.com
103. [103]
     The 80-question wall, tianpan.co, 2026-04-27. A practitioner blog on enterprise AI security questionnaires. Vendor tianpan.co
104. [104]
     Spanish authority issues guidance on agentic AI and GDPR, Covington (Inside Privacy), 2026-03-06. It summarises the AEPD's 71-page guidance. Secondary insideprivacy.com
105. [105]
     Agentic AI guidance from European regulators, TrustArc, 2026-08. It reports the Dutch AP warning and CNIL guidance. Single source trustarc.com
106. [106]
     COMPL-AI, arXiv 2410.07959, 2024-10. It maps the EU AI Act to technical benchmarks and states it is not legally binding. Primary arxiv.org
107. [107]
     COMPL-AI board, Hugging Face API. It shows the last change to the public board. Primary huggingface.co
108. [108]
     LatticeFlow releases COMPL-AI, SiliconANGLE, 2024-10-16. It describes LatticeFlow's EU AI Act framework. Secondary siliconangle.com
109. [109]
     "No one is holding the line", Aithos. It reports LARA results under EU law. Single source aithos.substack.com
110. [110]
     LARA leaderboard API, Aithos. It returns 3,681 judged leaderboard runs. Primary lara.aithos.org
111. [111]
     What is AIUC-1?, ElevenLabs. It describes the scope of the AIUC-1 standard. Vendor elevenlabs.io
112. [112]
     New research finds 80% of major AI platforms non-compliant with EU standards, Diplomacy AI press release, 2026-01-21. It announces the Janus AI Risk Index. Vendor business.mammothtimes.com
113. [113]
     Trust3 AI launches Trustscore, TechIntelPro, 2026. It describes an agent risk rating product. Vendor techintelpro.com
114. [114]
     AI Trust Mark, Nemko. It says direct model testing is not in the initial scope. Vendor digital.nemko.com
115. [115]
     EU-Agent-Bench, arXiv 2510.21524, 2025-10. It measures illegal agent tool calls under EU law. Primary arxiv.org
116. [116]
     EU-Agent-Bench repository, GitHub. It shows the date of the last push. Primary github.com
117. [117]
     South Korea's AI Basic Act: overview, Cooley, 2026-01-27. It describes impact assessments and procurement incentives. Secondary cooley.com
118. [118]
     Evaluating multi-agent systems (chapter 9 of this report), Ascending Labs, 2026-11-06. Our review of vendor documentation finds about 20 platforms that trace agent teams and no dedicated multi-agent testing vendor. Primary evals.study
119. [119]
     Debate or Vote (Choi, Zhu, Li), arXiv 2508.17536, UW–Madison, 2025-08-24. NeurIPS 2025 spotlight: majority voting accounts for most debate gains on seven benchmarks. Primary arxiv.org
120. [120]
     How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks, arXiv 2606.20695, UT Austin, Uber, 2026-06. Preprint: seven of ten designs report gains below a paired noise floor. Single source arxiv.org
121. [121]
     Towards a Science of Scaling Agent Systems (Kim et al.), arXiv 2512.08296, Google Research, Google DeepMind, MIT, 2026-04-08. Preprint: 260 configurations; independent agents amplify errors 17.2-fold. Primary arxiv.org
122. [122]
     GPT-6 Astra system card, OpenAI, 2026-10-30. Section 8.5: it cannot yet measure contact between agents of the same user in Codex. Vendor deploymentsafety.openai.com
123. [123]
     Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors, arXiv 2607.07368, UK AI Security Institute, 2026-10-12. Preprint: the first multi-agent control study; split attacks evade per-agent monitors. Primary arxiv.org
124. [124]
     Building multi-agent systems: when and how to use them, Anthropic, 2026-01-23. It puts a team's cost at 3–10× a single agent's tokens. Vendor claude.com
125. [125]
     Emergent Collusion in Long-Horizon LLM Agent Interaction, arXiv 2609.24967, Stanford, 2026-09-21. Preprint: collusion in 94% of mutual-verification trajectories across 10 models. Primary arxiv.org
126. [126]
     Agent evaluators, Microsoft Learn (Foundry), 2026-09-30. System evaluators apply to "the main orchestrator or the final agent". Vendor learn.microsoft.com
127. [127]
     Issue #243: multiple agents on the same telemetry, OpenTelemetry semantic-conventions-genai, 2026-06-04. Open as of 6 Nov 2026. Primary github.com
128. [128]
     State of AI Engineering 2026, Datadog, 2026-04. Customer telemetry: 59% of agent requests make one service call. Vendor datadoghq.com