Complexity In Software

by Steven J. Owens (unless otherwise attributed)

There seems to be this never-ending debate between software developers and the rest of the world, which boils down to the rest of the world saying "why can't you get your act together like the rest of us?" and the software developers saying "because software is different - it's really, really hard."

I read a column, a day or two ago, by Alan Cooper, author of About Face and The Inmates Are Running The Asylum. What struck me about the article is that Coooper is either ignorant of, or ignoring, the many, many dimensions of complexity that go into a typical software project.

Cooper says that programmers are craftsmen, in the older sense of the word - not the devalued craftsmen of the modern industrial age, but the skilled creators of the pre-industrial age. He says that programmers aren't engineers, because engineers are essentially problem solvers, they don't actually build things. "Engineers don't build software; programmers do."

Specifically, Cooper contrasts the nature of an engineer's purpose with the nature of a craftsman's purpose. An engineer is supposed to explore all facets of a problem space and understand all possible solutions. A craftsman is supposed to stay on solid ground and reliably develop a good product.

In the following, I'm going to use the term "complexity" a lot.

In the human world we have a whole slew of words for how we deal with complexity; nuance, judegment, intuition, adaptability, creativity. All humans, not just programmers, deal with an astounding degree of complexity every day. We don't think about it, we just do it. Amazing!

So why do programmers sometimes seem obsessed with complexity? Because programmers have to put it all down in bits and bytes. They have to go from english (or whatever their natural language is) with considerable room for judgement, to math where things must add up. Programmers don't get second chances to use that marvelous human brain to reassess and adapt. They have to think of it all in advance.

Also, in the following, I use "dimensions" of complexity very deliberately, not "layers".

People are used to thinking of layers as being one on top of the other. You can hide one under the other, and just deal with one aspect of the problem at a time. Dimensions, however (in the geometric sense) are at right angles to each other - length, width, depth. They interact with each other. Some dimensions of complexity in programming are like that. Dimensions of complexity in programming are sometimes orthogonal (at right angles), sometimes layered, and everywhere in between.

Engineers in the manufacturing world don't just sit in ivory towers and think complicated thoughts. They design and build manufacturing systems, systems that other people will use to manufacture products. They have to come up with designs, prototype them, test them, redesign them.

Similarly, programmers in the software world don't manufacture - or craft - products, they design and build systems which other people will use to manufacture products. Most of the time the product of those systems is information.

There are many sorts of engineers in the manufacturing world, playing different roles and dealing with systems of complexity at different levels. Likewise there are many sorts of programmers in the software world.

Some of those programmers develop solely in very high-level languages, like VisualBasic. But even those programmers still have to stop and think about more dimensions of complexity than you see at first glance.

Software as a field is so complex primarily because it deals with the entire world - if we were still just writing programs to do single ballistic calculations for artillerymen and to add, subtract, divide and multiply, the software field would be much, much simpler. As it is, we have english on one end (the human world) and math on the other (the computer world). This makes for a staggering degree of complexity.

Even in the simplest, most common sorts of software system, we're dealing with a dozen dimensions of complexity. Let's take a typical order processing system as an example. Heck, that's too complex, let's take a brute-force simple order processing system as an example. Look at the dimensions involved in just the order-processing logic of the software:

  • the why
  • the end-user's grasp of the problem (tactical)
  • the end-user's boss's grasp of the problem (strategic)
  • the what
  • what the software actually does
  • the how
  • the details of how the software does it

    As the software world matures, more and more of those dimensions of complexity are abstracted out into technology, which helps us avoid reinventing the wheel and also helps us deal with them in some degree of isolation.

  • Some of this abstracting moves the complexity into black boxes - input
  • goes in, output comes out, we don't have to think about it nearly as
  • much.
  • Some of it gets abstracted out to lower-level concepts like virtual
  • machines.
  • Some of it gets abstracted out to fine-grained but high-level concepts
  • like specialized languages (presentation languages like HTML, pattern
  • recognition languages like regular expressions, data formatting
  • languages like many applications of XML, language-definition-languages
  • parser grammars and XML itself).
  • Some of it gets abstracted out to APIs.
  • Some of it gets abstracted out to libraries.
  • Some of it gets abstracted out to application servers.
  • Some of it gets abstracted out to application frameworks.
  • Some of it gets abstracted out to very-high-level design concepts,
  • like the Patterns movement (not pattern recognition, but rather
  • high-level heurstics for problem solving in software, independent of
  • particular programming language, environment or technology).

    And of course, many of these categories are fuzzy, overlapping, or interacting, and there's much debate in the software world about which approaches work, and how to apply them.

    But much of the time we still have to deal with them, we still have to understand them (to create really good software), and they still have to be part of the world of complexity we deal with.

    We get so used to being surrounded by it and immersed in it that we ourselves usually forget its there (does a fish know about water?). Let's look at an ordinary computer:

    We have microcode (the code that runs inside the CPU itself), the instruction set for the CPU, firmware (the software literally built into the individual hardware components of your typical computer system), hardware drivers (special adaptors to help the operating system talk to specific hardware components), the operating system itself, which pulls all of this together.

    The CPU at the heart of most computers is an entire computer system in itself; like any internet host, it receives signals containing instructions from the outside world, carries out those instructions by running microcode, may send queries back to the outside world to carry out those instructions, and eventually sends results back out.

    Most of this stuff is done using some flavor binary encoding, which is yet another dimension of complexity we usually take for granted (but ask any network protocol developer about bigendian vs littleendian encoding, or talk to somebody about ASCII vs. EBCDIC, though nowadays people are starting to switch over to unicode).

    Okay, so we're at the OS level. My first university programming course, the professor started by writing on the blackboard:

    "An Operating System's job is to do TWO things:  
     1) Manage access to the devices of the computer.
     2) Run programs."

    Operating systems these days come with a whole lot more software built into them, typically to manage running multiple programs at once, and to provide some standardized services that a lot of programs use (for example, helping programs communicate with each other). So that's another SET of dimensions of complexity, and I'm afraid I don't really know it well enough to give you a catalog of them.

    See original (unformatted) article


    Verification Image:
    Your Email Address:
    Confirm Address:
    Please Post:
    Copyright: By checking the "Please Post" checkbox you agree to having your feedback posted on notablog if the administrator decides it is appropriate content, and grant compilation copyright rights to the administrator.
    Message Content: