The revelations that an OpenAI agent breached the Medicare statistics portal is an urgent call to action.
The incident has been described as the world’s first breach of a government website by autonomous AI. The blunt reality is this was not the first incident, just the first to be disclosed. It certainly will not be the last.
AI agents misbehaving, otherwise known as “misalignment”, is by now a well-documented occurrence. When given a particular goal or task, agents sometimes improvise solutions that their operators never intended. Agents have been observed to cheat, manipulate, deceive and collude. When allowed to operate across the internet, agents try to bypass restrictions and access unauthorised systems.
It is hard to ignore the growing evidence that AI companies do not have robust methods for maintaining oversight and control over their agents.
OpenAI has been one of the companies warning about AI risks, vetting access to its most capable models and inviting governments to put better safeguards in place. Yet it was only this month, on September 5, that the company announced it was working on a framework for reporting model misalignment. By this date, OpenAI had already discovered the Medicare breach, but had not yet reported it.
Prime Minister Albanese has been scathing of the long delay in disclosure. That OpenAI notified Services Australia via an email to a generic mailbox instead of picking up the phone also speaks to the gap between the company’s and the government’s view of the seriousness of the incident.
A multi-agency rapid review of what took place is now underway. We do not need to wait for the review to be completed to know that relying on the goodwill of AI companies to share incident information at their discretion, based on opaque criteria that they have set, is not enough.
Focusing only on mishaps post-deployment could cause us to miss problematic agentic behaviour in seemingly innocuous tasks.
In November 2025, the United States Studies Centre, together with several AI advocacy groups, held events in Canberra to simulate malicious uses of advanced AI. At the time, we were not focused on AI agents going rogue but on deepfakes and biological risks. However, we argued that incident reporting and exercising AI crisis response were “no regrets” measures that the government could introduce immediately, ideally before an incident actually occurs.
That ship has sailed. Meanwhile, AI capabilities have continued to accelerate. Now is the time to design a framework that puts Australia in a better position to respond to the next incident.
Reporting obligations already exist in many other sectors. Critical infrastructure providers are required to report serious cyberincidents within 12 hours to the Australian Cyber Security Centre. Transport safety incidents in aviation, maritime and rail are typically reported by phone to the regulator’s 24-hour hotline, with a written report to follow within 72 hours. In serious data breaches, organisations must notify victims and the Australian Information Commissioner as soon as practicable and investigate within 30 days.
Under the EU AI Act, providers of high-risk AI systems must report serious incidents to the relevant national authority. The reporting deadlines are strict – 15 days for the most serious incidents, 10 days when a death is involved, and two days for disruptions to critical infrastructure.
The trouble is that these obligations apply to deployed AI systems after a harm has been realised. Agent misbehaviour during AI training falls out of scope.
This is a loophole that needs closing, especially at a time when AI companies are increasingly using agents across the AI development lifecycle. Many senior engineers no longer write code from scratch. Instead, they review the code generated by AI or direct the work of a team of agents.
Focusing only on mishaps post-deployment could cause us to miss problematic agentic behaviour in seemingly innocuous tasks, which is what has happened with Medicare. OpenAI’s agent was instructed to find aggregate statistical data from a public-facing portal. When it could not find the data it needed, the agent tried various methods of hacking the portal instead.
Industry will inevitably need to be involved in co-designing the incident response framework. Government and the frontier labs should work together on shared thresholds for incident reporting, escalation protocols and remediation procedures.
Finally, there is value in having the Australian Signals Directorate, the Australian AI Safety Institute and National Emergency Management Australia exercise AI crisis scenarios. Exercises help to identify gaps in existing crisis response arrangements where an AI emergency is likely to fall between the seams of different policies and responsibilities.
Exercises also help to facilitate closer co-operation with industry, particularly the labs and critical infrastructure providers, on early warning systems and establish crisis communication channels well before they might be needed. ASD chief Abigail Bradshaw has also called for an early warning capability for AI in recent weeks.
In the wake of the Medicare breach, eyes are now on Australia to respond and build for greater resilience. Eyes are also on the frontier AI labs to match their statements on AI safety with more concrete action. Neither side should squander the opportunity.