New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review.