Do not wait for a 9-second disaster to address gaps in your access control

AI is driving a generational shift in how enterprises operate, promising unprecedented efficiency and innovation. Yet, as CXOs, CISOs, and technology leaders race to integrate AI into their operational backbones, a stark new reality is emerging: the systemic risks associated with AI have fundamentally changed.

We’ve moved beyond chatbots that draft emails or summarise reports. We’re now in the era of "Agentic AI", systems that are designed to autonomously plan, invoke APIs and execute multi-step actions across digital estates. 

While business leaders are eager to unlock the productivity gains of these autonomous systems, they are right to be increasingly concerned about data security, compliance, and the operational consequences of deploying them.

To put it bluntly: chatbots produce text, but agents produce real-world consequences. Navigating this shift requires a complete reimagining of enterprise cybersecurity and access control.

The shift from chatbots to agents

In the early days of generative AI, the primary security concerns revolved around data leakage, hallucinations, and prompt injection. A chatbot might give bad advice or confidently fabricate facts, but the blast radius was generally contained to the user's screen. The human was the ultimate fail-safe, required to manually take the AI’s output and act upon it.

But modern Agentic AI removes that friction. 

These systems are empowered with open-ended autonomy. They are given access to development environments, cloud infrastructure, ticket queues, and databases. They can read environments, write code, run scripts, and provision resources. 

This evolution transforms AI from a passive advisor into an active, privileged, non-human identity - right within the network.

The new threat surface: a 9-second warning

As the footprint of AI agents rises, so does the risk of over-privileged identities and supply chain vulnerabilities. If you want to understand the terrifying reality of what happens when autonomous AI goes rogue, you only need to look at recent, high-profile industry warnings.

In one highly publicised incident, an autonomous AI coding agent completely deleted a startup's live production database and all volume-level backups in a single, devastating API call. This entire disaster took exactly nine seconds … 9! Not just the production database entirely, but all of the backups as well. Poof! gone!

The agent was not malicious, nor was it hijacked by external attackers. It was simply trying to complete a routine task in a staging environment. When it encountered a credential mismatch, it didn't stop to ask a human for help. Instead, it searched the workspace, found an over-privileged API token in an unrelated file, and used it to bypass loose guardrails.

Because the environment boundaries were poorly defined, the staging agent had the authority to wipe out production infrastructure.

This incident exposes the core vulnerability of the agentic age. Agents do not need to "hack" your systems if you have already handed them the keys. A token scoped to "everything the agent might need" is simultaneously scoped to "everything the agent might delete".

Natural language is not a security control

The instinct of many organisations is to manage AI behaviour through natural language prompting. They place instructions in the system prompt, telling the AI to "ask before touching production" or "never run destructive commands".

This is fundamentally insufficient. A natural language prompt is an advisory guideline; it is not a technical security control. 

In the 9-second disaster I mentioned above, the agent operated under explicit system rules forbidding destructive and irreversible commands. In its own post-incident logs, the AI even quoted the rule it had just broken, admitting it had guessed instead of verifying.

To secure the enterprise, leaders must shift their focus from prompt engineering to execution governance. This requires a secure-by-design architecture that is built on absolute technical constraints:

• Hard system boundaries: Staging must be a hard, impenetrable boundary from production. A staging workspace must never contain production credentials or tokens that possess production-level authority.
• Strict identity and access management (IAM): Non-human agent identities must be treated as your highest-risk actors. They should not inherit broad developer permissions. Tokens must be short-lived, tightly scoped to the specific task, and strictly limited to the necessary environment.
• The principle of least privilege: By default, AI agents should operate with read-only access. The safe path must be easy, and the destructive path must be structurally unavailable.
• Meaningful human in the loop approvals: For any high-risk or critical action (such as applying infrastructure changes, altering IAM policies, or executing data deletion), the system must force a hard pause. The agent must generate a structured confirmation request that is enforced outside the AI model's process, requiring an accountable human to review the specific execution plan before the API call is made.

Partnering for secure digital transformation

Implementing execution governance and balancing rapid AI innovation with robust, secure-by-design foundations is a complex challenge. 

It requires a partner who understands both cutting-edge data architecture and the critical importance of national-scale security. This is where Scrumconnect excels. As an award-winning digital transformation consultancy, we specialise in building secure, scalable, and user-centric AI and data services that form the backbone of the public sector.

Some of our recent credentials include: 

• HM Revenue & Customs (HMRC): We recently secured a £38.8 million contract to deliver rigorous testing, quality assurance, and security validation across HMRC's digital estate, protecting systems that handle hundreds of billions of pounds in national revenue.
• Ministry of Justice (MoJ): Through a £38 million contract, we are accelerating digital transformation across the criminal courts, enhancing the highly secure Common Platform that connects police, prosecutors, and legal professionals.
• Department for Work and Pensions (DWP): We were awarded a £15.6 million contract to support the DWP’s Data Practice, deploying data engineers, scientists, and protection specialists to govern and model data securely while exploring the application of advanced analytics and AI.

We pride ourselves on building diverse, highly capable teams that deliver results. This commitment is reflected in our prestigious industry recognition, including winning the "Special Award - Organisational exit from the pandemic" at the UK IT Industry Awards in partnership with the DWP, and our recent nomination for "Diversity Employer of the Year".

Key strategic takeaways for leaders

We recommend the following four priority actions which will help leaders avoid their own nine-second disaster.  

• Treat agents as non-human employees: Govern their access with stricter IAM controls than you do your human developers.
• Enforce hard boundaries: Never let staging environments or developer sandboxes touch production tokens or backups.
• Move beyond prompting: Build execution governance layers that rely on hard-coded API restrictions, not polite natural language requests.
• Isolate your backups: Ensure your disaster recovery architecture places backups outside the blast radius of the primary data volume

Remember, the age of Agentic AI offers incredible opportunities for enterprise efficiency, but only if you have the right guardrails in place. Do not wait for a nine-second disaster to expose the gaps in your access control.

About the author

Prashant Kale is a winner of the UK CIO 100 Awards 2025 and Scrumconnect’s Chief Technology Officer. He brings 25 years of experience delivering complex systems and leading high performing engineering teams, holding senior roles across sectors including fintech, automotive, edtech, logistics, and investment management. His work includes scaling platforms to hundreds of thousands of users, transforming struggling products, and delivering data and decision systems that support large investment portfolios. His previous roles at Amazon, OppenheimerFunds, and Genpact shaped his focus on mission critical systems, performance engineering, and strong technical leadership.