Saturday, August 14, 2021

Conservation of Complexity

Software always involves abstraction. I'd be happy to hear clarification from genuine information theorists, but I think there's a "convervation of complexity" in using a toolset. (Or in trying to solve a problem with laws and regulations.)

For instance, if you have a server farm of identical systems, it's nice to have some simple procedures for standing up a new system automatically, or wiping one whose system disks failed and reinstalling. There are lots of tools for that.

Such a tool will let you collect all the common stuff together, and let you customize the details for different servers with different purposes. Compute servers, disk servers, DNS, wiki, database servers--we have patterns for them, and customizations for the patterns. The same database server software, loaded in the same place, with the same admin--and each instance adds different users and tunings and backup times.

You can get some huge gains in productivity at first. Then those pesky corner cases bedevil you. E.g. a new software version isn't backwards compatible with the old. You need both old and new. Do you break commonality and add a new class to your management tool, or add lots of "if"s into your scripts? The former means you have to manage changes in two places at once; the latter makes the code complex and bug-prone. You get complexity either way.

Containers and clouds are all the rage in computing, and something like Kubernetes is wonderful for making certain classes of service resilient and easy to scale. Not everything fits in that model. I've listened to the people trying to make that work. I think I'll be retired by the time disillusion with that magic bullet sets in. I also noticed that the big clouds aren't 100% reliable in availability or in integrity (parler). It's nice to have hardware of your own in the back room, and a manager you can collar without waiting for the ticketing system.

I've heard stories of corporate deciding to use a new wonder-software that requires redoing workflow, and losing all the putative productivity savings in the expense of the changeover.

Abstractions always leave out bits of the real world. When you try to stretch it too far the real world always wins. sometimes people die

1 comment:

Christopher B said...

Not a theorist but I've also been in the trenches (CS BS in 1983, did application programming for a while, then systems engineering for IBM mainframes for about 30 years) and you're dead-on with your observation about abstractions. Every system is designed with a certain set of assumptions about real world trade-offs and procedures, and it's exceedingly rare that those are actually documented upfront. You usually find out when you're trying to fit the square peg of a requirement into the round hole of capability. Who would have imagined that an on-call scheduling system would *not* support switching duties on a given day each month?