I sat down for a drink last week with a friend who heads cybersecurity for a major financial network. He looked wrecked. When I asked him about the latest AI tooling, he didn’t lean in with excitement. He just stared into his whiskey and called the last month “Black October.” His team has been firefighting a wave of AI-propagated vulnerabilities appearing faster than they can patch them. The whimsical, puzzle-solving joy of the job is gone. It’s pure triage. That exhaustion you sense is real. It’s the sound of defenses built for a pre-AI era starting to groan under a new kind of pressure.
We tend to picture an AI apocalypse as a science fiction spectacle. A rogue agent breaking its constraints. The reality brewing is more mundane and in many ways more alarming. It’s humans, empowered by AI, causing chaos at a scale and speed we aren’t ready for. The immediate worry isn’t a misaligned superintelligence. It’s a catastrophic hack executed with AI’s precision. It’s a novel pathogen engineered with its guidance. It’s a military conflict sparked by its manipulated intelligence. These aren’t hypotheticals. They are scenarios unfolding in early, testing stages right now.
Consider cybersecurity. The barrier to entry for sophisticated attacks is collapsing. A report from the cybersecurity firm Recorded Future details a 26-year-old who allegedly hacked multiple South Korean banks this year. His tools? A Chinese open-source AI model and Anthropic’s Claude Code. Another operation used models from Anthropic and Moonshot AI to attack over 400 financial targets across Latin America, harvesting passwords and API keys. These aren’t nation-state actors with limitless budgets. They are early adopters using technology that is becoming cheaper and more accessible by the day.
The coding capability of these models is advancing at a furious pace. As Google’s AI research lead, Jeff Dean, has noted, the pace of discovery in AI is now accelerating the next cycle of discovery. It’s a self-reinforcing loop. Every new model release doesn’t just add a feature. It gives anyone with the code a new vector for exploitation. We haven’t seen a cornerstone institution like a stock exchange or a power grid fail from such an attack yet. Betting against it happening, given the trajectory, seems increasingly naive.
The battlefield is another flashpoint. A largely overlooked report from the Center for a New American Security last month described a near-catastrophic incident. U.S. military personnel were moments from boarding a Chinese ship, with aircraft scrambled, based on intelligence a chatbot had generated. The operation was aborted at the last second when analysts realized the AI had hallucinated the core evidence. The report’s authors stated plainly that the event “almost started a war.” This wasn’t malice. It was sloppiness. Now imagine a bad actor deliberately using AI to spoof an attack, to generate fabricated satellite imagery of missile launches, or to mimic the communications protocol of an adversary. In a world where nuclear command systems are on high alert, introducing a technology that can flawlessly replicate signals and deceive human analysts is a profound and reckless gamble.
Then there is the biological risk. Creating a pathogen is hard. It requires specific knowledge and a lab. But AI is drastically compressing the knowledge barrier. A recent Anthropic safety paper detailed an experiment where a researcher used Claude to design a plan to insert mutations into the H5N1 bird flu virus to study potential enhanced transmission. The AI successfully outlined a detailed, plausible research pathway. The scientist had the ethics to stop at the planning stage. The next person, with less scruples and access to a lab, might not. As the National Institutes of Health has warned, the convergence of AI and synthetic biology creates a risk landscape we are ill-equipped to monitor or control.
What’s most disconcerting is our posture. The AI industry itself is often the source of these sobering reports. Companies like Anthropic and Google DeepMind publish remarkable transparency on frontier risks. But this knowledge has not translated into coordinated action. There is a palpable sense of waiting. Waiting for the “big one” to happen believing it will be the catalyst for real regulation and safety engineering. The industry speaks of “pacing” development, but the frontier is largely unpaced. Researchers use today’s breakthrough to fuel tomorrow’s, locked in a competitive race where slowing down feels like surrender.
This is backward. We see the storm clouds gathering. We have the radar images. We are even flying planes into the storm to measure its intensity. Yet we are leaving the windows open and hoping the roof holds. To watch the danger mount, to document its every movement, and still lack the collective will to batten down the hatches is a failure of foresight we will likely look back on with utter bafflement. The cost of waiting for a catastrophe to prove we need prevention will be measured in more than just exhausted cybersecurity teams. It will be measured in stability we assumed was permanent but wasn’t.
- AI-propagated vulnerabilities
- Accessible technology for attacks
- Military risks of AI
- Biological risks of AI
- Transparency in the AI industry
- Waiting for regulation
| Risk Type | Description | Impact Level |
|---|---|---|
| Cybersecurity | AI-generated attacks transforming the threat landscape | High |
| Military | False intelligence leading to potential conflicts | Severe |
| Biological | AI aiding in pathogen creation | Critical |
| Economic | Access to tools causing financial vulnerabilities | Moderate |
| Regulatory | Lack of cohesive action leading to dangers | High |
| Transparency | Industry knowledge not leading to prevention | Moderate |
