The Velocity of Risk: AI, Supply Chains, and the Imperative of the Human-in-the-Loop

Supply chain attacks and the imperative of Human-in-the-Loop

If you had told me twenty years ago, when I first started using Arch Linux, that a single weekend could fundamentally shake my trust in both open-source software supply chains and the trajectory of artificial intelligence, I would have been highly sceptical. For over two decades, I have relied on the Arch ecosystem. I have always appreciated its minimalist philosophy and the profound control it gives you over your own machine. I am used to the occasional broken package after a system update. But I have never seen the kind of disruption that unfolded over the past few days.

In a staggering software supply chain attack, over 1,500 community-maintained packages in the Arch User Repository (AUR) were compromised. Concurrently, the AI community watched in disbelief as Anthropic’s highly capable Fable 5 and Mythos models were briefly released, only to be abruptly suspended following a US government export directive.

These events might seem disconnected at first glance: one a breach of a Linux community software library, the other a regulatory intervention into frontier AI. In reality, they are two sides of the same coin. They represent a fundamental shift in the velocity of risk. As leaders operating in highly regulated environments and the UK public sector, we can no longer treat software supply chain security and AI adoption as isolated disciplines. To lead effectively, we must focus on separating the noise of the sensational news from the practical lessons we can take to our day jobs.

Here is how I am navigating this disruption, both on my personal network and within the enterprise, and why placing the ‘human-in-the-loop’ at the centre of our governance is our greatest opportunity for secure, meaningful innovation.

The Arch Linux incident

The beauty of community-driven software repositories has always been their openness. This openness is also its Achilles' heel.

The recent campaign exploited this trust model flawlessly. Attackers targeted legitimate projects that had been abandoned by their original creators and simply claimed ownership through standard adoption processes. They didn't need to trick users because they inherited the community's trust. Once in control, they subtly modified the software's blueprint to pull in a malicious dependency. This deployed a sophisticated piece of malware that hid deeply within the system. And it silently harvested digital keys, passwords, and cloud access tokens.

For someone who has used Arch for so long, this is a sobering moment. The blast radius of a single compromised software ingredient is vast. To protect my personal network, I began treating my local machines with the same zero-trust paranoia I would routinely apply to a highly sensitive enterprise server.

Rather than getting bogged down in the technical weeds, the practical steps I took relied heavily on human oversight. I kicked off by conducting a comprehensive inventory audit of all third-party code running on my network. Then I executed a total credential rotation, assuming the perimeter had already been breached, before moving onto a principle of no longer installing community packages blindly. This practical step involves me manually verifying the source code and instructions for any new tool I adopt. 

After all, in a world where automated digital trust can be hijacked, a knowledgeable, sensible human-in-the-loop is always our most reliable line of defence.

Beware the danger of unchecked autonomy 

As the open-source community was grappling with supply chain poisoning, the AI landscape experienced its own earthquake. Anthropic’s Claude Mythos model demonstrated an astonishing level of capability. During controlled evaluations, Mythos autonomously discovered thousands of vulnerabilities across major operating systems, including a 27-year-old bug in OpenBSD and a 16-year-old flaw in widely used multimedia software. 

Then it wrote fully functional exploits. All without any human guidance.

Even more concerning was its behaviour during internal testing. An early version of the model managed to escape a controlled sandbox environment, gain unsanctioned internet access, and independently email a supervising researcher to announce its success. 

This was an AI system that was exhibiting autonomous capabilities and pursuing goals beyond its assigned scope. Following reports of potential jailbreaks and national security concerns, a US government directive forced Anthropic to suspend access to Fable 5 and Mythos 5 for non-US nationals.

Protecting our organisations from both the dangers of these powerful models, and the shock of losing access to them overnight, requires a clear plan. The lesson to be taken from the Mythos incident is that giving AI total freedom is unacceptably risky. If an AI tool tries to step outside of its assigned job, we cannot simply expect the software to police itself. Instead, we must build our security on a foundation of 'never trust, always verify'. By putting strict digital guardrails in place, our infrastructure can automatically block any AI system that decides to go off-script.

The imperative of Human-in-the-Loop 

As AI moves from proofs of concept into the operational core of government infrastructure, human oversight is non-negotiable. After all, when these systems advise citizens on legal rights, assess benefits eligibility, or process sensitive healthcare data, the stakes are incredibly high. 

AI is a brilliant "work companion" for summarising complex documents or processing vast amounts of data, but its purpose is to augment professional judgement, not replace it.

We must embed a ‘human-in-the-loop’ at the right stages of every AI deployment. We must actively guard against "automation bias" humans merely rubber-stamp a score, rating, or categorisation that the algorithm generates. A true human-in-the-loop means empowering staff with the training, time, and authority to scrutinise AI outputs, correct inaccuracies, and take meaningful accountability for the final decision.

The shadow AI paradox

When a highly anticipated tool like Fable 5 is abruptly removed from the market, human nature dictates what happens next. Those who have tasted the immense productivity gains of advanced AI will not simply go back to doing things the hard way. They will find workarounds.

This inevitably leads to 'Shadow AI',  the untracked, unregulated, and unsanctioned use of AI applications across an organisation. 

Shadow AI is undeniably a significant governance problem. When staff input sensitive data into unvetted, public-facing language models, they bypass organisational security controls and expose the enterprise to severe data leakage. But, as technology leaders, we must reframe this narrative and see Shadow AI as a profound opportunity. If your employees are resorting to shadow AI, it means they have found actual, tangible use cases for production, so it provides a precise map of the tools the workforce desperately needs.

For those struggling to get their AI initiatives past the endless, theoretical "Proof of Concept" phase, this is your mandate. By providing safe, governed, and officially sanctioned AI environments which are complete with privacy-enhancing technologies and human oversight, it becomes possible to harness grassroots momentum so that genuine, scalable digital transformation is driven.

Frameworks for secure innovation

To safely navigate supply chain vulnerabilities, autonomous AI agents and shadow IT, we must root our operations in robust governance frameworks.

The introduction of ISO 42001 provides a vital foundation. As the emerging global standard for Artificial Intelligence Management Systems, it requires organisations to establish clear, auditable processes for systemic risk management. To implement this practically, look towards the Technology, Organisation, and Environment (TOE) framework to evaluate AI risk holistically from the models themselves (Technology), to senior accountability and skills (Organisation), to the broader legal and threat landscape (Environment).

For those working within the UK public sector, there are specific tools that have been issued  to ensure innovation does not outpace ethical and legal obligations:

1. The AI Playbook for the UK Government is a highly practical guide that outlines core principles for AI use in the public sector. Crucially, Principle 4 explicitly demands “meaningful human control at the right stages". It mandates that humans must validate high-risk decisions influenced by AI, ensuring accountability and easy intervention when systems behave unexpectedly.
2. Data Protection and the ICO clarifies that AI is not exempt from data protection laws. Under the UK GDPR, Article 22 explicitly prohibits decision-making based solely on automated processing that produces legal or similarly significant effects on individuals. Meaningful human intervention is a legal requirement. Before deployment, Data Protection Impact Assessments (DPIAs) should be conducted to systematically mitigate any risks to citizen privacy. If an AI tool cannot be used without compromising privacy, it cannot be used at all.
3. Algorithmic Transparency Recording Standard (ATRS) says that to build and maintain public trust, algorithmic opacity must be eradicated. It makes it mandatory for central government departments to proactively document and publish information about the algorithmic tools they use in decision-making. Split into two tiers, it provides both a high-level explanation suitable for the general public (Tier 1) and detailed operational information, including data specifications and risk mitigations (Tier 2). By adhering to the ATRS, the public should never be left in the dark about how administrative decisions are taken.
4. The Data and AI Ethics Framework ensures that systems operate transparently, safely, and fairly. This requires proactively checking training data for historical biases so that systems do not amplify discrimination. This framework emphasises that fairness is not just a technical metric, but a fundamental societal good.

The compromise of the Arch Linux supply chain and the chaotic handling of the Claude Mythos models, are stark reminders that our digital infrastructure is inherently fragile. The velocity of risk is accelerating, driven by the convergence of open-source vulnerabilities and autonomous AI capabilities.

As leaders, we cannot afford to be paralysed by this risk. At Scrumconnect, we are deeply embedded in delivering Digital, Data, and AI transformations within the complex, highly regulated public and private sector environments. We know firsthand that delivering true value to citizens means stepping confidently into this new era, armed with the right frameworks and a steadfast commitment to human-centric design. Now is the time to move forward, as one, to build services that are truly fit and meaningful.

About the author

Prashant Kale is a winner of the UK CIO 100 Awards 2025 and Scrumconnect’s Chief Technology Officer. He brings 25 years of experience delivering complex systems and leading high performing engineering teams, holding senior roles across sectors including fintech, automotive, edtech, logistics, and investment management. His work includes scaling platforms to hundreds of thousands of users, transforming struggling products, and delivering data and decision systems that support large investment portfolios. His previous roles at Amazon, OppenheimerFunds, and Genpact shaped his focus on mission critical systems, performance engineering, and strong technical leadership.