This OWASP guide popped up on my radar this week and, yes, it’s about AI. And yes, it’s entirely predictable. But what appeals to me at the moment is its predictability amidst the nondeterminism of LLM rakestepping. Catastrophic outcomes in these complex systems are foreseeable not just from today, or the day this Adversa post was published, but at least from 1984. It was in 1984 that sociologist Charles Perrow published “Normal Accidents: Living With High-Risk Technologies.” Normal Accidents had nothing to do with artificial intelligence, yet seeing how it’s being deployed today, the book now has everything to do with it. Perrow studied major industrial accidents across much of the twentieth century and isolated some important insights on unexpected catastrophic failures inevitable enough to be called Normal Accidents:
- The system is complex.
- The system is tightly coupled.
- The system has catastrophic potential.
In the agentic systems we see proposed and being implemented before us, certainly complexity plays an integral role - the dirty little secret of LLMs is that to make one useful, especially for a specialized expert task, you’re dealing with multiple layers of LLMs with varying levels of autonomy. It’s the sausage being made behind that single pane of glass most AI products pretend to be.
We then turn to tight coupling - essentially, complex systems producing outputs that must occur in a specific order, such as a multi-stage chemical treatment process. It is the anticipated sequence - in Perrow’s words, the invariant sequence - where B must follow A, because that is the only way to make the product - that defines tight coupling. Think about the sub-tasks each Agent is charged with; pre-prompt hardening against injection attacks, shifting tone and scope of the LLM response, providing expectations to shape system output. Above that and the primary agent doing the task, you have multiple other systems working to evaluate, validate, and re-shape output before it’s pushed to the surface agent, who relays it to you. Should those multiple subsystems interact in varied ways or orders, the output is necessarily - perhaps catastrophically - affected.
Catastrophic potential is mostly self-evident, but let us take a specific example: the modern Security Operations Center, or SOC. Perrow’s book provides multiple corollary environments - think a Nuclear Power operations center full of sensors, monitors, and potential alerts. Or the cockpit of a commercial airplane, which had seen much more automation in the decades prior to 1984 and provided starkly relevant examples of alert and attention issues at critical moments. Indeed, we see SOC failures in some of the biggest hacks on record, where alerts are missed or disregarded, leading to major systemic damage.
So in the SOC we have a complex, tightly-coupled system with catastrophic potential. “The essence of the Normal Accident,” Perrow wrote, is “the interaction of multiple failures that are not in a direct operational sequence.” That is, system components interacting in sequences and ways not only unexpected, but “incomprehensible” during the incident, often leading to much worse outcomes.
And what do we do, 42 years after Normal Accidents’ release? We add a complex, relatively tightly-coupled system of agents to a complex, certainly tightly-coupled system with catastrophic potential called the Security Operations Center. And not only that, but a system of agents fundamentally empowered by their own nondeterministic nature.
“What distinguishes these [system component] interactions,” Perrow wrote, “is that they were not designed into the system by anybody; no one intended them to be linked. They baffle us because we acted in terms of our own designs of a world that we expected to exist - but the world was different.”
In the rush to the AI/Agentic SOC, expect many Normal Accidents.
Podcasts
- ChinaTalk - Richard Danzig on AI and Cyber - Danzig’s published “Artificial Intelligence, Cybersecurity, and National Security: The Fierce Urgency of Now” over at RAND, and sat down for a good interview with the ChinaTalk folks. I may not agree with all points raised, but the conversation is a very well-informed and thoughtful one.
Articles
- CERT Polska - Energy Sector Incident Report - If you’re reading this list, you’ve probably already seen this, but linking in case that’s untrue. Probably the best of the recent resources around the Polish energy grid incident, and worth becoming familiar with. Also good writeups by Kim Zetter here, here, and here.
- UK Defence Journal - Iranian-linked Scottish accounts fall silent again - Social media accounts that posed as supporters of Scottish Independence once again fell silent amidst the Iranian internet blackout. Fun little detection, if not necessarily a fine one.
- TechCrunch - Fintech firm Marquis blames hack at firewall provider SonicWall for its data breach - I expect this to go nowhere, but it’s certainly an interesting push that the likes of Ivanti, Fortinet, and Palo Alto will also be eyeing nervously.
- Greynoise - -f Around and Find Out: 18 Hours of Unsolicited Telnet Houseguests - Telnet? What year is it?!?! All kidding aside, Greynoise is making some fascinating moves lately, and as much of an AI skeptic as I am, hrbmstr’s experiments with AI analysis show some real value there.
- APNIC - What we learned from 63,000 attacks in 12 days on APNIC Honeynet sensors at University of Dhaka - “In just twelve days, our sensor was hit 63,247 times by 4,262 unique source IPs, including five unique IP addresses from Bangladesh. Fourteen of those led to malware download attempts. The time to first attack was less than one hour, and we averaged ~5,270 attacks every single day!” - Not groundbreaking, but another eye-opening bit of research showing with data just how dangerous the internet can be, packet-wise. Good work and writeup on their part.
- Resecurity - Cyber Counterintelligence: When “Shiny Objects” Trick “Shiny Hunters” - Resecurity published some great coups between December and January, and have gone on my must-read feeds list as a result.
Research Papers and Reports
- GTIG - No Place Like Home Network: Disrupting the World's Largest Residential Proxy Network - Good work by Google taking down the IPIDEA proxy networks, but a lot remains to be done.
