Loss of control is among the most discussed risks from frontier AI. It is also one of the hardest to analyze concretely. Most existing frameworks treat it as a model problem, one that begins with deception, scheming, or behavioral drift inside the system itself. This paper starts from a different premise. When a frontier lab deploys an AI coding agent internally, with write access to its own codebase, infrastructure, and evaluation pipelines, the risk lives in the whole sociotechnical system, which includes the developers who prompt the agent, the pipelines that gate its output, the monitoring that watches it, and the organizational policies that configure all of the above.
Aviation, nuclear, and chemical processing learned decades ago that accidents in complex systems rarely trace back to a single broken component. They emerge from interactions between components that are each working as designed. The paper applies three methods built on that insight to a generic frontier-lab coding agent scenario, reconstructed entirely from public disclosures. STECA audits published governance frameworks for completeness. STPA models how a control structure can fail to keep a system in a safe state. FRAM examines how ordinary operational variability can erode safeguards without anything having locally failed.
Each method surfaced risks that model-focused evaluations would miss. The STECA analysis found that published safety frameworks describe critical control actions, such as pausing model use or assessing catastrophic risk, without naming anyone responsible for executing them, and that the authority to expand deployment scope is specified while the counterpart authority to contract it is not. The STPA analysis showed how monitoring latency and procedural delay can render a correct intervention ineffective, arriving only after the harm is done. The FRAM analysis identified four patterns through which a well-designed control structure degrades in ordinary use: safeguards drift in a correlated direction under shared productivity pressures, model upgrades outpace monitor calibration, agent-authored code gradually becomes the baseline against which future agent output is judged, and human reviewers anchor on the agent’s own explanation of its work.
The recommendations follow from the findings. Frontier AI developers should pair model-focused evaluations with systems-level hazard analysis of their internal deployments. Regulators should recognize that static framework documents cannot show whether controls remain calibrated over time, and consider requiring periodic operational indicators such as monitoring coverage, alert-to-intervention latency, and the balance between agent-generated code volume and human review capacity. Because some of these metrics are security sensitive, the paper proposes a two-tier regime in which aggregate indicators are published openly and full operational data goes to a designated auditor under confidentiality.
The methods themselves required little adaptation, and the same procedures apply unchanged inside any specific developer, with only the inputs substituted. The analysis that matters next is the one run with operational data.
Back to top