Saturday, August 11, 2012

The opposite of simple is not complex

Studying biology or economics, one finds organisms, ecosystems, and economies that are more than the sum of their parts. Somehow many interacting agents with limited information produce increasing organization, creating amazing complexity out of relatively simple components.
In computing, if we want to harvest this potential for surprise, see results this interesting, we have to write complex systems.

This doesn't mean we want to write complicated systems.

Wait, isn't that contradictory? Complex systems that are not complicated?
Ah, but that is the whole point of complexity theory. 

Herbert Simon described a complex system as being composed of hierarchy and near-decomposable parts.[1] Just as an economy is composed of many humans making decisions independently, a piece of software can be composed of many parts that can't see inside each other. This preserves the potential for producing complex behavior, solving a complex problem. Yet, each part within the system may provide a simple abstraction.

The difference between complex and complicated - or, as Rich Hickey calls it, complected - is intertwining the different modules in the system. An application may consist of many uniform modules, each of which knows about the inner workings of the other -- complicated. Or it may consist of even more modules, each of which interacts only with the abstractions exposed by the others -- simpler, and yet with potentially complex results.

The key is breaking the software into independent modules, each of which says to the others "I don't know how you do what you do, and I don't care." Each module or component can be developed by a separate team with different coding standards or in a different language. Uniformity is compromised *gasp*. No single architect understands all parts of the system. Strict top-down, God-like orchestration is sacrificed. Instead, component interactions are abstract. Teams interact at higher levels, without knowledge of each others' code.

Each component handles versioning, security, releases, backwards compatibility - everything, in its own way. Reuse is at the component level, at the level of release, not at the class or function level.

A system based on independent components like these probably contains more code than a functionality-equivalent system that is orchestrated top-down. That's okay. It has potential to support much greater solution complexity.

A fact of software development: 
For every 25 percent increase in problem complexity, there is a 100 percent increase in complexity of the software solution. That's not a condition to try to change (even though reducing complexity is always a desirable thing to do); that's just the way it is. [2]
This is true because every feature we add potentially impacts every feature we already support. The increase in solution complexity is combinatorial, not linear. The way to combat this is to reduce the complectedness of our software. Stop intertwining all the bits -- a coherent architecture is more efficient in lines of code, but it is inherently limiting. It is limited by what the God Architect can hold in his head.

Break the software into simpler components. Some of these may be open-source components, which someone has to learn well enough to configure and implement. Others will be custom. Break them off into teams and give those teams autonomy. Let each team implement its piece in the simplest way possible for that particular problem.

Meanwhile, the systemwide architects don't know the particulars of each solution. In fact, they should not know the particulars. That would allow them to base solutions on implementation details. Don't let that happen! Raise the level of abstraction. Components must interact with abstractions, not with each other. Knowledge of inner workings of other modules is a negative.

Jeff Bezos forced this strategy at Amazon around 2002 by decree: all teams will interact via service interfaces only; they can use whatever technology they want; and all interfaces must be externalizable, ready to be exposed to the outside world. The result? The largest online bookseller became the largest vendor of cloud computing. Did anyone predict that ten years ago?

When you and I interact, our brains do not interact directly. Rather, we both interact with the physical world. I say something, you hear it. Abstractions on both sides reduce the granularity of the interaction, but maintain the independence of our individual brains. No mind-melds allowed. While this seems limiting, observation shows that amazing unpredictable structures - nations, communities, economies - emerge from these limited interactions.

Complicated systems are limited in growth. Complex systems have even greater potential for growth than the designers of the components conceived. Keep your components simple and independent, and there is no end to the problems we can solve.

[1] Complexity, A Guided Tour, by Melanie Mitchell, chapter 7
[2] Facts and Fallacies of Software Engineering, by Robert L. Glass, p. 58


  1. VERY nicely put!

    Where I struggle with this is in user-interface development.

  2. Fantastic article.

  3. Hy jessie
    That's exactly what Udi Dahan and others have been talking about for a long time. True SOA style architecture embodies your reasoning and even provides answers in how to compose user interfaces in such an architecture.
    The beauty with such decomposed systems is not only that you can idependently develop it or use different languages you can also apply JFHCI (just freakin hard code it). This has huge business advantage: you can invest in quality where you have good knowledge and are pretty sure you got it or hard code parts where you are uncertain but want to gain insights into the market.

    Nice article!

  4. Neat comparison of "complex" vs "complicated" and how it applies to software systems. It's a great and logical expansion/application of loose coupling principles.

    Also worth noting, the biology analogy you cite is a great to not only show the possibilities, but the dangers of the "simpler components" approach. While biological systems are indeed a very complex sum of their simple parts, the system came to be through evolution, not design, and there's some serious systematic consequences to this. Vestigials, instinct-driven decisions, and uhh, male nipples?

    Software systems that develop through evolutionary means (especially those that are "componentized" by autonomous teams) can become a whole new kind of complex.

    It starts with: do we have a webservice that formats addresses? Oh great, err, it doesn't work exactly the way we want... ugh, that team is backed up. We'll just write our own.

    Years later: so.... which of the 6 address formaters should we use?Or worse, when the business rules change ("street names now precede house numbers")... oy.

    So, I think it's good to have a balance of automony and knowledge of *some* of the inner workings to allow for planning.

  5. Hey, male nipples are awesome!

    In the example given - does inner-working knowledge help with that at all?
    The sin here is that some team added to their piece of the system an additional responsibility. What if the address formatting module was open-source? It has a team of maintainers but anyone can contribute. That's one way to deal with the resource-allocation issue, which is really what triggered this bifurcation of responsibilities and violation of DRY.

  6. In starting up a business, colorado llc would like to know how those software helped the business to improve and to gain such success