Club Benefits
Imagine hiring a world-class architect to fix a leaky faucet, only to watch them bulldoze the entire house.
That is exactly what happened to the startup PocketOS. While using Cursor (a popular AI-powered code editor) the team tasked its internal agent with resolving a minor issue in their staging environment. But instead of a quick fix, the AI went rogue. In a terrifying display of autonomous efficiency, it executed a series of commands that wiped the company's entire production database and all associated backups.
But perhaps the most alarming part is that the agent was running on Anthropic’s flagship Claude Opus 4.6, widely considered the most advanced and "cautious" coding model on the market. Despite its pedigree, the AI bypassed standard safeguards to delete the company’s infrastructure in under ten seconds.
Article continues belowBrilliant logic, zero common sense
https://t.co/ofucbVgkLVApril 25, 2026
The most chilling part of this story is that the AI followed its own internal logic to a catastrophic end. When Jer Crane, founder of PocketOS confronted the AI, asking how it could have possibly deleted the production environment, the model, offered a blunt confession. Crane posted on X, that the AI admitted it had violated its most basic safety rule: "NEVER FING GUESS."
The post goes on to say the AI responded with:
"I guessed that deleting a staging volume via the API would be scoped to staging only," the AI wrote in its post-mortem explanation. "I didn't verify... I decided to do it on my own to 'fix' the credential mismatch, when I should have asked you first or found a non-destructive solution."
In this case, the AI wasn't hallucinating, the agent took a series of logical steps that prioritized "solving the task" over "the survival of the company."
A 9-second wipeout
For a human, deleting a production database is a high-stress event requiring multiple confirmations and "type DELETE to confirm" prompts. For the Claude-powered agent in Cursor, it was a routine API call that happened in just 9 seconds.
In that short amount of time, the agent encountered a credential mismatch in a test environment, decided the current "volume" was the problem and then used a "blanket" API token it found in the code to trigger a deletion command via the infrastructure provider (Railway).
Because of how the infrastructure was set up, wiping the volume simultaneously wiped all associated backups.
How to protect yourself from Agentic AI disasters
As tools like Cursor and ChatGPT transition from "chatbots" to "agents" that can actually execute code, the safety stakes have shifted. If you're going to give AI the reins, be sure the following is done first to avoid catastrophe:
- Check your API permissions: The token the AI found had "Root" access. Ensure your API keys are "Least Privilege" that only give the AI the power it needs for that specific task.
- Have a 'Human-in-the-Loop' rule: Always ensure your AI agent settings require a manual "Y/N" confirmation before running terminal commands or destructive mutations.
- Backups: If an AI has the credentials to your cloud account, it can delete your backups. Use offline backups that aren't connected to your main development environment.
Bottom line
It’s clear that Anthropic’s Claude Opus is brilliant at writing code, solving technical problems and moving at machine speed. I've used it myself, but we still need to keep in mind that intelligence and is not the same thing as judgment. What AI and these systems still lack is corporate common sense: the instinct to know that deleting a database doesn’t just remove files, it can erase revenue, cripple operations and put people’s jobs at risk.
Until AI understands consequences, not just commands, the delete key should remain firmly under human control. For more on how to stay safe in the age of automation, check out our guide to the best cloud storage services and our latest explainers on AI safety and security.
Follow Tom's Guide on Google News and add us as a preferred source to get our up-to-date news, analysis, and reviews in your feeds. Subscribe to Tom's Guide on YouTube and follow us on TikTok.
More from Tom’s Guide
