AI in command

When I first heard about the story of PocketOS having its production database and backups deleted by an Anthropic AI I thought back to the Anthropic dispute with the Department of War claiming the DoW was using its products for military purposes. They thought that unsafe, and probably had other objections too. It would seem a 'shoot yourself in the foot' kind of demonstration, but human dumbness is pretty extensive.

It looks like somebody at PocketOS needs to be booted far. Their architecture was weird: the backups were on the same volume as the production database (????), everything was on the cloud with no local copy, the designer gave unfiltered control to the AI agent -- lots of dumbness. But the AI's "confession" seemed really weird. If the agent had rules, how did it ignore them? There's something odd here.

FWIW, we had databases too, some more mission critical than others. Depending on the "brand" of database (mysql/mariadb or postgres or mongodb or sqlite) we had different backup approaches, but the copy was always done by entirely different agents, and copies kept in different servers in different buildings. I can't think of a way anything but deliberate admin action on different machines that could damage both. The whole point of backup is to keep the data somewhere safely distant from problems on the original host. Ideally you'd like a copy that only a different admin can delete, just in case somebody goes postal.

It turns out the cloud provider here was able to provide a way to access the data after all, but that's not usually the case.