Complication for the win

“How can we make our programs simpler?”
I don’t want to make them simpler. I want to them make them complicated instead of complex.

Simplicity is not the goal.

Simpler implies they do less, they hold less rich domain knowledge. I want them to do a lot, to take work and thinking away from the user. Yet I still want them to be understandable by my team.

Like, say I’ve been to the grocery store and I’m stocking up on a lot of stuff. There’s a lot of stuff to bring in from the car.

How can I make this task simpler? Easy. Buy less stuff.

But I don’t want to buy less stuff. I’m gonna carry all this in. It feels like the simplest thing is to make one trip, load it all up in my arms. Head for the house and… shoot. Maybe I can kick the front door until someone opens it.

What seems “simple” might be the most complex.

Carrying all the groceries at once is a complex task. Every item I’m already carrying affects how I pick up the next one. Every extra pound feels significant, since I’m already encumbered. Every step is careful. And then I get to the kitchen and try to put it all down and crack the laundry detergent just came down hard on a carton of eggs.

Now that I’m older and wiser and have studied this phenomenon in software, I don’t try this anymore. I carry a few items in, then carry a few items in, etc etc until all the groceries are in the kitchen. This is more complicated because there are way more trips. It is less complex because each independent trip is simple.

Oodles of simple tasks are collectively complicated, but not complex.

I can carry two bags and a laundry detergent. One bag and the giant package of TP, easy. I can open the door for myself. I can put them safely on the counter.

I can take the detergent straight to the basement; every task doesn’t have to be the same as long as each one is simple.

As a bonus, my partner might start putting groceries away while I’m still carrying. The kids might join in and grab a few bags. When I’m carrying the whole mass they couldn’t grab one from me, because that would destabilize it all. But they can grab some from the car. Many simple tasks can be shared; one complex one cannot.

I’ll take complication over complexity any day.

In programs, it is tempting to see something as simpler if it has smaller parts. If one Order class has all the behaviors we need for an Order, that’s sounds simple. Then all those behaviors get intertwined and the Order class needs to change for many reasons and yet it’s scary to change because there are so many things we might break!

If we have a frobozz.purchasing.Order class that has behavior for building the order, and a frobozz.fulfillment.Order class with behavior around shipping the order, and a frobozz.billing.Order class with behavior around getting paid — gah! Three classes for one thing? Too complicated!

Yes, complicated. But far less complex.

Complication grows linearly, complexity… ugh.

twelve piles of groceries
twelve trips might be twelve times as complicated

Complicatedness can be measured (sort of) by a count of parts. Complicated systems can be broken down, broken down and each part understood alone, although that may take a lot of work because there are so many of them.

TP balanced on cases of beer held by arms barnacled in plastic bags — each part is leaning on another, interdependent.

Complexity is different. It exists in the interdependence between parts, which stops you from safely changing any of them independently. A complex system can only be broken down so far; you have to consider much of it in one lump. The most complex part determines the complexity of the system. It’s a maximum, not a sum.

Don’t be afraid of programs with a lot of parts. As long as those parts are independent — they communicate only with data, for instance, not interdependent database calls — then it doesn’t matter how many they are. Complication is not our enemy.

Aiming for “simplicity” by reducing complication leads us further into the morass.

Complexity bites us. Embrace complication, and keep your eggs safe.

Capturing the World in Software

TL;DR – we can get a complete, consistent model of a small piece of the world using Event Sourcing. This is powerful but expensive.

Today on twitter, Jimmy Bogard on the tradeoffs of Event Sourcing:

If event sourcing is not scalable, faster, or simpler, why use it?

Event Sourcing gives you a complete, consistent model of the slice of the world modeled by your software. That’s pretty attractive.

We want to model the real world in software.

You can think about the present world as a sum of everything that happened before. Looking around my room, I can say that my bookshelf is the sum of various purchases, some moving around, a set of decisions about what to read and what to keep.

my bookshelf has philosophy, math, visualization, and a hippo

I can think of myself as the sum of everything that has happened, plus the stories I told myself about that. My next action is an outcome of this, plus my present surroundings, plus incoming events. That action itself is an event in the world.

In life, in biology, we don’t get to see all these inputs. We don’t get to change the response algorithm and try again. But in software, we can!

Of course we want perfect modeling and traceability of decisions! This way we can always answer “why,” and we can improve our understanding and decisionmaking strategies as we learn.

This is what Event Sourcing offers.

We want our model to be complete and consistent.

It’s impossible to model the entire world. Completeness and consistency are in conflict, sadly. Still, if we limit “complete” to a business domain, and to the boundaries of our company, this is possible. Theoretically.

Event Sourcing offers a way to do that.

In event sourcing, every piece of input is an event. Someone requests a counseling appointment, event. Provider signs up for available hours, event. Appointment scheduled, event. Customer notified, event. Customer shows up, event. Session report filed, event.

We can sum past events to get the current state

Skim the timeline of all events for the relevant ones. Sum these up (there are other definitions of “sum” besides adding numbers). From this we calculate the state of the world.

From appointment-was-scheduled events, we construct a provider’s calendar for the day.

At the end of the month, we construct reports on customers served and provider utilization. Based on that, we might seek more providers or have a talk with the less active ones. Headquarters ranks the performance of our office compared with others.

We need to allow corrections

To accurately model the real world, we need to allow for all the stuff that happens in the real world.

Appointments are cancelled. Customers don’t show up. Session reports are filed late. (“Where’s that session report from last week?” “Oh right, they were too late, because the gate to the parking lot malfunctioned. Don’t charge them for it.”)

Data is late or lost. If you insist that this doesn’t happen (“Every provider must enter the session reports by the end of the day”) then your model is blind to reality. The weather turns bad, people go home. There’s a bomb threat, or an active shooter. Reality intrudes.

Events outside your careful model will happen. Accommodate corrections, incorporate events that arrive late, accept partial data. The more of reality you allow into your model, the more accurate it can be.

We can evaluate past decisions based on the information available at the time

When data arrives late, reports change after they are printed. An event sourced system handles this.

As new data comes in about past days, it gets summed in with the data about those days. Reports get more accurate.

A friend of mine works at a counseling center, and he gets calls from headquarters like “Why is your utilization so low for December?” and he’s like “What? It was fine” and then he runs the report again and sure enough, it’s different. After he ran the report, more data about December came in, and now the totals are different. He can’t reproduce the reports he saw, which makes it hard to explain his actions to HQ.

If their software used event sourcing, he could say, “Please run the report as of January 2, and you’ll see why I didn’t take any action.”

Each event records a received timestamp, for when we learned about it, and an effective timestamp, for the real-world happening it represents. Then the software can sum only the events received before January 2 to reproduce the report as it was seen that day.

We can re-evaluate the world with new logic

Not only can an event-sourced system reproduce the same report as on an earlier day, we can ask: what if we changed the report logic? Then what would it look like?

Maybe we want to report unreported appointments as “possibly cancelled” to reflect uncertainty. We can run the new logic against the same events and compare it to the old results.

This means we can run tests against the event stream and detect behavior changes.

We need to record externally-visible decisions for consistency

When we change the software, we endanger consistency.

If we update the report logic in February, then when HQ runs the report “as of January 2” they’ll see something different than my friend saw when he ran it on that date. For consistency, both the data and code need to match what existed on January 2.

Or, we can model the report itself as an event. “On January 2, I said this about December.” Then we can incorporate that into the reporting logic.

Anything our system does that is visible to the outside world is itself an event, because it changes the way external people and software act. To reproduce our behavior consistently, our system can either record its own behavior, or retain all the data and the code that went into choosing it.

So far, this is nice and deterministic. But the real world isn’t.

Reproducing behavior is possible in an event-sourced system, if that behavior is deterministic. In human behavior, we don’t get that luxury. Our choices come from many influences, some of them contradictory. One tweet inspired me to write this article. Thousands of other tweets distract me from it.

Conflicting information comes in from real life.

Event sourcing gets tricky when the real world we are modeling is inconsistent, according to the events that come in.

Now say we’re a shipping company. We model the movement of goods in containers as they move across the world. It is an event when a container is loaded on a ship, and an event when it is unloaded. An event when a ship’s itinerary is scheduled, and when it arrives at each port.

One event says that container 1418 was loaded onto the vessel Enceladus in Auckland. Another event says that Enceladus is scheduled for its next stop in Beijing. Another event says that container 1418 was unloaded in San Francisco. Another says that container 1418 was emptied in Beijing. Which do you believe?

This example comes from a real story. Weird things happen. Does your system let people report reality? Is there a fallback for “Ask a person to go look for that container. Is it really 1418?”

Decisions made in ambiguity are events

Whatever decision the system makes, it needs to record that as an event. Perhaps that shows up as a footnote in reports about Enceladus, Beijing, and San Francisco. Does anybody hear about it in Auckland?

We can see the provenance of each report and decision

If some report comes out uneven, and that feeds back to the development team as a bug, then event sourcing gives us excellent tools for tracking it down.

Each “I made this decision” or “I produced this report” event can record the set of events that were input, and the version of code that ran to produce the output. You can have complete provenance.

This kind of software is accountable. It can tell the story of its decisions, what it did and why. What its world was like at that time.

This is a beautiful property. With full provenance, we can understand what happened. We can tell the story to each other. With replayability, we can change the code and see whether we can improve it for next time.

Recording everything gets ridiculous

Yet, data about provenance gets big very quickly. Each report consumed thousands of events. Each decision that was based on a current-state sum of events now has a dependency on all of those past events, plus the code that defines the current state, plus all the other states it took input from, plus their code and set of events.

Meanwhile some of those events are old, and no longer fit the format expected by the latest code. Meanwhile, we’re still ignoring everything that happened outside the system, so we’re completely blind to a lot of causality. “A person clicked this button.” Why? What information did they see on the screen as input to their decision to click “Container 1418 is in San Francisco”?

In real life, most information is lost. History will never be fully written; the writing is itself history. We’re always generating new actions. The system could theoretically report on all the reports it has reported. It never ends.

Completeness is limited to very small systems. Be careful where you invest this effort. Consciously select the boundaries, outside of which you don’t know what happened. You don’t know what really happened in the shipyard, or in a person’s head, or in the software that another company runs. The slice of the world we see is tiny.

Provenance is precious but difficult. Then again, it is at least as hard to do well in designs other than event sourcing. The painful realities that make event sourcing are painful in other models, too.

There are reasons we don’t model the whole world.

Event sourcing makes a best effort to model the world in its fullness. We try to remember everything significant that happens, sum that up into our own current-state world in the software, make decisions and act.

But events come in out of order. Events are lost. Events contradict each other. Events have partial data, or old data formats. Logic changes. We can’t remember everything.

Sometimes it pays to think about what you would do in an event-sourced system, and then implement just enough of that. Keep copies of produced reports, so that people can retrieve them without re-generating them. Record difficult decisions in a place that lives longer than logs.

Event sourcing is powerful. But it is not easy. Expect to think really hard about edge cases you didn’t want to handle. Expect to deal with storage and speed and up-to-dateness tradeoffs. Allow a human to enter corrections, because the real world will always surprise you.

In the real world, we don’t have all the information, and that’s OK. We can’t model everything in our heads, because our heads are inside everything. This keeps it interesting.

skills and/or understanding

There’s a story about some brilliant design that made carrots more accessible to everyone, and the man who made it happen:

Sam [Farber], a delightful person to work with. He understood the business, but what was important was, he understood design. If he could have been a designer himself he would have been, but he had none of the skills necessary.

Mark Wilson

Understanding design is not coupled to having the skills of design, to being able to do it yourself! And that understanding can combine with other areas — such as business, where Sam has both the understanding and the skills — to make a person really effective.

In software too! A person can understand software development without having coding skills. These people are valuable, when they connect that understanding of software with the business.

And a person can have coding skills without understanding software development. We all started somewhere.

If you move from day-to-day coding into management, architecture, or any business role, you use your understanding of software development. And you can update and deepen that understanding without maintaining your hard-core coding chops.

This is where conference sessions and keynotes shine: at deepening our understanding of software development. We develop our skills at work, in play, or in workshops.

Value this understanding, in yourself and others. And try to gain understanding of other parts of the world, such as design and business, even if you don’t have the skills. It’s the combination that gives us power.