Micheal Bailey in Engadget: What happens when AI starts self-replicating?

"I think [self-replicating models] are in the future. I think they're real. I think we'll see them," says Michael Bailey, a professor at the School of Cybersecurity and Privacy at Georgia Tech. "Whether they'll take on the exact form and shape, and have the same consequences, that [Anthropic] is pitching is a completely open question."

Bailey has been studying internet worms since the early 2000s, and more recently, botnets. In 2017, he co-wrote Understanding the Mirai Botnet, which Anthropic cites as part of its own research. Mirai allowed a group of cybercriminals to create a large botnet of IoT devices they later used to carry out a DDoS attack against Sony's PlayStation Network. In the process, they ended up affecting Dyn, one of the largest DNS providers in the US. In turn, the attack broke much of the internet in 2016, knocking Amazon, GitHub, PayPal, Reddit and Twitter offline for many users across the US and parts of Europe. The financial fallout of the attack was significant, with Sony estimating it lost about $2.7 million in net revenue.

"What Mirai showed is that you could compose these botnets — and have them be useful — out of everyday devices: cable routers, IP-enabled cameras, these kinds of things," says Bailey. "We thought those devices weren't useful to an adversary, either as targets or as sources of computation for launching attacks. In fact, they're actually quite effective at doing those kinds of things."

A decade later, Anthropic argues AI worms could represent a meaningful evolution of Mirai and other hard-coded botnets like Kimwolf because they could figure out unfamiliar devices on their own. Bailey says adaptability itself isn't new. Mirai already worked across "heterogeneous" devices, from routers to IP cameras running on different chip architectures. But that adaptability was finite: each new architecture required human work, at minimum recompiling the malware. What's novel, Bailey says, is the prospect of AI automating that work, though he doesn't think Anthropic demonstrated it "in a compelling way."

As a result, Bailey suggests more capable AI models will "redistribute" who does the work of creating botnets. "It's not going to do new stuff," he says. "It's going to change the economics between attackers and defenders." While Anthropic showed that a model could hop between devices and set up inference on each, it did so in an emulated network where the "exploit" involved was trivial. Additionally, Bailey takes issue with Anthropic's self-replicating claim.

"The framing is somewhat of a misnomer, because they're doing things like sharding. Not that that's bad, but it's not like a worm or a virus. It's not full propagation," he says. Instead, Bailey argues a real-world AI botnet might work more like a dropper, a type of malware that gets a foothold on a device and installs other code. Rather than copying itself wholesale onto every machine it compromises, the botnet would install a small coordination tool that takes instructions from elsewhere in the network.

"The idea that the malware itself is self-replicating, like existing internet worms, where I'd copy an exact copy of my code to the new architecture and it would start running a new agent — that's not really what's happening here," says Bailey. He also notes worms can be polymorphic, meaning they can change both their signature and behaviors. For an AI botnet to do that, he says, it would need to retrain its own models, something Anthropic didn't attempt. The closest it came was showing that copies of a model could pass notes to their successors and get better at setting up inference over time.