An independent, Assurance 2.0-based review of Google DeepMind's public safety case for Gemini 2.5 Pro finds gaps that materially affect what the case can support, and sets out concrete recommendations for how external review of frontier AI safety cases should be done.
In 2025, Google DeepMind published something rare in frontier AI: a structured safety case arguing that Gemini 2.5 Pro could not cause severe harm through scheming, even if it were misaligned. This paper reports what happened when an independent team spent three months testing whether that argument holds.
Safety cases are structured arguments, supported by evidence, that a system is safe enough to deploy. In aviation, nuclear, and medical devices they are never accepted on the developer’s word alone. Independent review is what turns a self-assessment into something regulators and decision-makers can trust, and frontier AI has no established equivalent yet. The review team of six, with backgrounds spanning systems safety assurance, AI governance, machine learning, and evaluations, applied the Assurance 2.0 framework over roughly four person-months, with limited engagement from DeepMind itself.
The case earns real credit. It is unusually explicit about its argument structure, its dependence on empirical evaluations, and some of its own possible defeaters, and DeepMind’s decision to publish it enabled exactly the scrutiny this paper provides.
The review nonetheless surfaced concerns that materially affect what the safety case can support. The case never defines severe harm, leaving the property being assured unspecified. It does not describe the boundary of the assured system, so it is unclear whether the claim covers the bare model or the full deployment stack of scaffolding, tools, and memory that shapes what the model can actually do. Internal deployment could mean Google DeepMind or the whole of Google, with 185,000 employees and products serving billions, and exposure differs enormously between the two readings. Weighing the evidence, the team found the evaluations moderately confirm that Gemini 2.5 Pro fails DeepMind’s specific proxy tasks, but only weakly confirm the deployment-relevant incapability the argument rests on. No argument is offered for ruling out sandbagging, where a model strategically underperforms on the very tests designed to detect its capabilities. History also pressures a load-bearing premise. The claim that humans under ordinary oversight cannot cause severe harm sits uneasily beside the 2024 CrowdStrike incident, in which a single faulty update passing through ordinary controls disrupted 8.5 million systems.
From this experience the paper draws concrete recommendations for both sides of a review. Developers should publish safety cases in structured form, with defined system models, hazard logs, criticality analyses, and explicit validity conditions, and should grant trusted reviewers controlled access to evidence that cannot be made public. Reviewers should form their own risk pathways and criticality assessments before reading the developer’s, a discipline that guards against inheriting the author’s blind spots.
The gaps identified reflect the current state of the field rather than a failing unique to DeepMind. They are also the strongest argument for making external review routine before safety cases are used to justify deployment decisions.