While studying for the SC-100 Cybersecurity Architect exam, I came across a thought-provoking LinkedIn post from Abby Shakelford that raised a question I do not think enough organizations are asking yet:
What happens when an AI system causes a significant financial loss, but there was never a traditional cyberattack?
The scenario is simple.
An organization grants an AI agent legitimate access to business systems. The AI uses valid credentials, accesses authorized data, and performs actions it has permission to execute. However, an unintended decision results in substantial damage. There is no malware, compromised account, ransomware event, or unauthorized access. Just an AI making a bad decision while operating exactly as designed.
As Abby noted, that creates an interesting insurance challenge. Depending on policy language, insurers could reasonably argue there was no cyber event at all. The loss is real, but it may not fit traditional cyber insurance definitions.
While I found the insurance discussion fascinating, my SC-100 studies immediately took me in a different direction:
The Security Architecture Perspective
One of the controls included in the Microsoft Cloud Security Benchmark is AI-5: Ensure human-in-the-loop. That control recognizes that organizations should design AI applications with appropriate human oversight and governance controls rather than assuming fully autonomous operation.
In my view, this becomes especially important when AI agents are granted the ability to perform actions that could affect finances, security configurations, customer data, or production systems.
Abby's insurance question made me think about AI-5, but it also highlights another important control in the Microsoft Cloud Security Benchmark: AI-4: Apply least privilege for agent functions. If an AI agent can create a million-dollar loss, was it granted more authority than it should have had in the first place?
For decades, organizations have implemented approval workflows for:
- Financial transactions
- User provisioning
- Privileged access changes
- Firewall modifications
- Production deployments
- Data deletion requests
Nobody would allow a junior administrator unrestricted rights to modify critical infrastructure without oversight.
So why would we allow an AI agent to do so?
AI Does Not Need Zero Gates
One concern I have when listening to discussions around autonomous AI is the assumption that the goal is eventually to remove humans entirely from the decision-making process.
From a security architecture perspective, that should rarely be the objective.
Instead, we should be asking:
- Which actions can be fully automated?
- Which actions require approval?
- Which actions require multiple approvals?
- Which actions should never be delegated to AI?
The answer depends on risk.
If an AI agent is scheduling meetings or summarizing documents, the consequences of a mistake are relatively minor.
If that same AI can:
- Execute financial transactions
- Modify security configurations
- Provision privileged users
- Delete data
- Approve purchases
- Interact with customer accounts
Then the risk calculation changes dramatically.
The Real Lesson from the Insurance Debate
The insurance industry is rightfully asking who pays when AI creates damage.
Security architects should be asking a different question:
The insurance industry is debating who pays after an AI mistake. Security architects should be asking what controls should have prevented the mistake before it happened.
Many discussions around AI risk focus on the model itself.
- Will it hallucinate?
- Will it leak data?
- Will it be manipulated through prompt injection?
Those are important concerns.
But as AI evolves from generating content to performing actions, governance becomes equally important. AI systems are increasingly acting as agents that interact with applications, data sources, APIs, and business processes. Microsoft specifically identifies risks associated with AI applications invoking functions and external integrations, which can lead to unintended actions if not properly controlled.
The problem is no longer simply, Can the AI generate a bad answer?
A Cybersecurity Architect's View
One of the most valuable lessons I am taking away from my SC-100 studies is that security architecture is rarely about trusting technology completely.
It is about designing systems that remain safe even when technology behaves unexpectedly.
That philosophy applies equally to cloud platforms, identities, privileged access, and now AI.
If an AI agent can create a million-dollar loss with no human review, the bigger question may not be whether insurance should cover the damages.
The bigger question may be whether the architecture should have allowed the action at all.
Final Thoughts
Abby's post raised an excellent risk-management question about liability and insurance coverage in the age of autonomous AI.
Studying Microsoft's AI security guidance while preparing for SC-100 has reinforced a lesson that feels increasingly relevant: the right answer is not only human oversight, but also least privilege for the AI agents themselves.
The future may absolutely involve AI agents performing more work on our behalf.
But from a security perspective, the highest-risk actions should continue to have a human in the loop.
Not because AI is inherently unsafe.
But because good security architecture assumes that every system, whether human or machine, can eventually make a mistake.