Rethinking AI governance in the age of autonomous agents

It’s no secret that organisations are rapidly deploying AI to streamline their operations, enhance user experiences, and drive unprecedented levels of efficiency. What is probably less well known, however, is that the dial has been quietly moved beyond advisory AI chatbots that merely draft text, and into the era of Agentic AI where the autonomous systems are empowered to plan, invoke APIs, and execute multi-step actions across digital estates.

That shift has now suddenly, and very publicly, been thrust into the public limelight. And the fallout, while contained, is alarming. 

Hackers successfully manipulated Meta's recently deployed AI customer support chatbot to hijack high-profile Instagram accounts. By exploiting the AI assistant, attackers quickly seized control of prominent Instagram including the global cosmetics retailer Sephora, a US Space Force Chief Master Sergeant and the archived White House account used during the Obama presidency.

While these are recent news stories and our professional community will await the specific technical details, we must separate the sensationalisation of the headlines from the precautionary steps we need to take so that such incidents do not occur within our own organisations. 

To Meta's credit, they identified the vulnerability and swiftly rolled out an emergency patch. However, this incident serves as a vital case study for any CXO or public sector leader: when we delegate sensitive administrative capabilities to autonomous AI, traditional security perimeters are no longer sufficient.

The confused deputy moment when AI becomes a privileged user

To understand how to protect our own organisations, we must briefly look at how this breach functioned. 

The attackers seemingly used VPN’s to spoof their geographic locations, making it appear as though they were logging in from the victim's hometown to evade automated security alerts. They then opened a chat with the Meta AI support assistant and simply instructed it to link a new, attacker-controlled email address to the target profile.

Due to a logic flaw, the AI complied and sent the security verification code to the hacker's email, bypassing multi-factor authentication precautions entirely, and handed over the password reset link. Security experts refer to this as a textbook 'confused deputy' attack, a scenario where a helper system with high privileges is tricked into exercising those privileges on an attacker's behalf.

For leaders managing highly sensitive public services or financial data, this underlines a critical reality. An AI agent is not just software; it acts as an active, privileged, non-human identity operating directly within your network. 

If an AI has the authority to alter records or bypass 2FA, then attackers no longer need to 'hack' your infrastructure. They just need to figure out how they can persuade your AI.

Why natural language is not a security control

The instinct of many organisations is to manage AI behaviour through natural language prompting, telling the system to "always verify identity" or "never run destructive commands".

This is fundamentally insufficient. A natural language prompt is an advisory guideline, not a technical security control. Because language models process system instructions and user inputs as part of the same data stream, they cannot reliably distinguish between a legitimate request and a malicious instruction.

As I have warned before, AI agents become fundamentally unsafe when they possess the "lethal trifecta": access to private information, internet connectivity and the ability to receive an untrusted instruction. Relying solely on polite AI prompts is obsolete against dynamic, reasoning agents that can be socially engineered.

A blueprint for execution governance

To safely harness the productivity gains of AI, leaders must shift their focus from 'prompt engineering' to Execution Governance. This requires a secure-by-design architecture built on absolute technical constraints:

1. Strict identity and access management (IAM): Non-human agent identities must be treated as your highest-risk actors. They should operate under the strictest principles of least privilege, with access tightly scoped to specific tasks.
2. Hard system boundaries: Staging and development environments must be an impenetrable boundary from production. The safe path must be easy, and the path to destructive or highly sensitive actions must be structurally unavailable to the AI acting alone.
3. Meaningful human-in-the-loop: For any high-impact action (eg altering core user credentials, modifying sensitive records, or applying infrastructure changes) the architecture must force a hard pause. An accountable human must review and approve the specific execution plan outside of the AI model's process before the API call is made.

‍Strategic AI governance for the C-Suite

For CXOs navigating these challenges, technical controls must be paired with robust, top-down organisational governance. To transition from ad-hoc deployments to formalised frameworks, leaders should implement the following strategies:

1. Establish an AI governance board: Create a dedicated board of senior leaders,  compliance officers, and technical experts to oversee AI adoption, assess risks, and ensure alignment with the organisation’s business objectives.
2. Maintain a live AI Systems inventory: You cannot secure what you cannot see. Leaders must maintain a comprehensive register of all deployed AI systems, their purposes, and data flows. For the UK public sector, this aligns directly with adopting the Algorithmic Transparency Recording Standard (ATRS), ensuring public trust and clear accountability.
3. Mandate data protection impact assessments (DPIAs): Before deploying any AI that processes personal data, mandate thorough DPIAs to assess privacy risks and ensure strict legal compliance. This is a non-negotiable step for the public sector and regulated bodies.
4. Adopt formal AI management systems: Adopting international standards, such as ISO/IEC 42001, provides a structured, globally recognised framework to manage the unique risks of AI. It ensures continuous improvement, transparency, and ethical alignment in your AI operations.

At Scrumconnect, we’ve and continue to partner with our clients such as DWP, DFE, HMCTS, Home Office, HMPO, HMRC which are highly complex, national-scale organisations to safely navigate such transitions. By deploying data protection specialists alongside data scientists, we ensure that their exploration of advanced analytics and AI is scaled within a rigorously governed, secure-by-design framework.

Securing the future of service delivery

The integration of Agentic AI into complex workflows presents unparalleled opportunities for enterprise and public sector efficiency, but only if we have the right guardrails in place.

As we build the next generation of services, our eagerness to innovate must be matched by robust execution governance and board-level oversight. As we separate the sensationalism of recent breaches from the tangible lessons they provide, we can implement the absolute technical constraints and recognised governance standards required to protect our users, maintain public trust, and confidently embrace the agentic AI revolution.

About the author

Prashant Kale is awinner of the UK CIO 100 Awards 2025 and Scrumconnect’s Chief TechnologyOfficer. He brings 25 years of experience delivering complex systems andleading high performing engineering teams, holding senior roles across sectorsincluding fintech, automotive, edtech, logistics, and investment management.His work includes scaling platforms to hundreds of thousands of users,transforming struggling products, and delivering data and decision systems thatsupport large investment portfolios. His previous roles at Amazon,OppenheimerFunds, and Genpact shaped his focus on mission critical systems,performance engineering, and strong technical leadership.