Wednesday, May 22, 2024

Oopsies

Putting your processing "in the cloud" has some nice scalability features. For example, the student registration program needs only a few CPUs most of the year, but for a few weeks it's nice to have a few hundred instances running, without having to have those systems idle the rest of the year.(*)

Putting your data "in the cloud" has nice redundancy: it can be in several different datacenters. If an earthquake takes out one, there's another datacenter that has it too, or maybe a third. Yes, you pay a bit more than managing your data yourself, but the tools are there to access your data from anywhere.

Until it isn't there. It seems Google deleted the account of an Australian pension fund, and all of its records went poof. No backups--pointers to backups went poof along with the account.(**)

an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription. This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.

Until it does. Luckily, at $135 billion, "UniSuper (is) a big enough company that, if something goes wrong, it gets the Google Cloud CEO on the phone instead of customer service."

Fortunately a wise planner at UniSuper spent the extra money to have a backup cloud service from an independent vendor, but for a couple of weeks they weren't able to send out pension checks. (I hope that wise planner wasn't fired for spending too much money.)


(*) You pay extra for speed. If you want your calculations done sometime this year, the CPU time is cheap. If you need to crunch a lot of numbers for results in time for the conference next week, expect to pay more.

(**) That is a security feature. If you could get access to data that used to belong to somebody else's account, you could do nasty things with that information. So, no account means no access. A new account will almost certainly not be exactly the same--I wouldn't design it like that, anyhow.

No comments: