A lot of rules

A Swede, on American Football: “Are there any rules? It looks like they stand in two lines, someone throws the ball backwards, and then it’s a big pile.”

Me: “141 pages, last I checked. It takes a lot of rules to look like there aren’t any.”

Later, people talked about error messages. There are so many different ones! … or there should be. Actually they all fall back to “Something went wrong” when the server responds with non-success.

Was it a transient error? Maybe the client should retry, or ask the person to try again later. Was it a partial failure? Maybe the client should refresh its data. Was it invalid input? Please, please tell the person what they can do about it. In all supported languages.

The happy path may be the most frequent path, but it is one of thousands through your application. As software gets more complicated, the happy path becomes unlikely. (99.9% success in a thousand different calls has a 37% happy path, if the person gets everything right.) What makes an app smooth isn’t lack of failure, it’s accommodation of these alternate paths. Especially when people are being humans and don’t follow your instructions.

Each error is special. Each chance to smooth its handling is precious. People rarely report confusion, so jump on it when they do.

This alternate-path code gets gnarly. You may have 141 pages of it, and growing every year.

It takes a lot of rules to make an app “just work.”

Every action has two results (Erlang edition)

Every action has two results: a set of side effects on the world, and the next version of ourselves.

I learned this from Erlang, a purely functional yet stateful programming language. Erlang uses actor-based concurrency. The language is fully immutable, yet the programs are not: every time an actor receives a message, it can send messages to other actors, and then return a different version of itself, instantiated with different state.

Here’s an example from the tutorial in the Erlang docs:

%%% This is the server process for the "messenger"
%%% the user list has the format [{ClientPid1, Name1},{ClientPid22, Name2},...]
server(User_List) ->
    receive
        {From, logon, Name} ->
            New_User_List = server_logon(From, Name, User_List),
            server(New_User_List);
%%% ...
   end.

This defines a server process (that is, an actor) which receives a logon message. When it does that, it builds a new list of users including the one it just received and all the ones it knew about before. Then it constructs a new version of itself with the new list! (That’s implicitly returned from receive.) The next message will be received by the new server.

It’s like that with us, too. Today I made coffee, with a side effect of some dirty cups and fewer coffee beans, and a next version of me that was more alert. Today I checked twitter, with a side effect of nothing observable, and a next version of me ready to write this post. Now I’m writing this post, which will have side effects with unknown consequences, depending on what y’all do with it.

This works in our teams, too. Every task we complete changes the world, and it changes us. Maybe we add tests or implement a feature. In the process, we learn about the software system we participate in. Did we do this as a team, or will we catch each other up later? Is changing the software more safe or harder than before?

When “productivity” measures focus on externally-visible outcomes, sometimes the internal system is left in a terrible state. Burnout in people, “technical debt” in code, and a degeneration of the mental models that connect us with the code we care for.

The consequences of our work matter now. The next version of us matters for the whole future, for everything after now.

Let’s reason about behavior

When we learn math, geometry, and logic in school, we’re always talking about things that are holding still. The element is in the set or it isn’t. The angle is acute or obtuse or right or we can’t know. Things are related or not. A thing has a property, or doesn’t.

Code can have properties. A function has side effects, or doesn’t. Data is mutated, or never. An API call can be idempotent. Whole programs can have properties and we like to reason about them.

Yesterday talking with Will Larson on >Code (episode 142), he pointed out that once we move beyond a single process, especially when we go to microservices and have oodles of processes running around, we don’t get to talk about properties anymore. Instead, we have behaviors. He said:

Properties, you can statically analyze.
Behaviors, you can verify they happen.

A call to one program, or even a series of calls, can be transactional. Once you’re in a distributed system, not a thing. You can talk about how the system behaves.

Rein pointed out that in distributed systems, properties are incredibly expensive. Guarantees like all-or-nothing transactions, exactly-once delivery, consistency are never perfect in the real world, and the closer you choose to be, the more you pay in money and latency. Coordination is expensive.

In addition, can we get better at verifying and reasoning about behaviors?

Will pointed out that fault injection is a way to verify behaviors. That makes sense: psychologists learn a lot about the way we think from a few people with localized brain injuries. Emitting and querying events is another way.

Then how do we reason about behaviors? Systems thinking helps. Will recommends Donella Meadows’s Primer as a start. (I loved that book too.) Also, the social sciences have been studying behaviors forever. Maybe their methods, like Grounded Theory, can help us.

We’re people, right? We have behaviors. If we can get better at naming and reasoning about them, maybe we can get better at being people. It could happen.

Other people’s messes

There’s a funny thing, that when I walk into the kitchen and there’s sunscreen on the counter that I left there before yesterday’s bike ride, there’s a plate that I put to soak on the counter and there’s a book about Klimt that I had hoped to read with my coffee, this is fine.

It doesn’t feel messy. I almost feel a comforting sense of continuity with my past self.

But when other people leave crab rangoons and cups and bowls and mail and washcloths lying around, ugh! mess!

It’s somehow different when I have the story behind it.

Code is like this too. If I remember writing it, then I know this nutty interface name was a placeholder. This ratty console.log helped me debug a tricky crash. I’m in the process of refactoring to use the new functions, but the old ones are still around some places.

When that’s anyone else’s code, it looks like trash.

This tells me: if the code is my own toy, and I’m likely to be back to it before I forget who I was when I wrote it, then all those intermediate states are fine. They might help me get back into this groove, even.

But if the code is shared, or if I won’t be back in it later this week, make small changes completely. One at a time, in series, so that each commit leaves the code clean. Put the sunscreen on and put it away. Wash the plate and put it in the dishwasher immediately. Take my coffee upstairs to drink with the book.

Sometimes I wish I lived alone, so I’d only have my own mess to deal with. I don’t want to code alone, though, except on toy projects. I do wish I lived with people who cleaned up after themselves. I resolve to make smaller messes in our shared code, one at a time.

Mostly we orient

Observe, Orient, Decide, Act. This is the OODA loop, first recognized in fighter pilots and then in the Toyota Production System. It represents every choice of action in humans and higher level systems: take in sensory data, form a model of the world, choose the next action, make a change in the world.

At least in fighter pilots, and in all our daily life, most of this is automatic. We can’t help observing while we are awake. We constantly decide and act, it is part of being alive. The leverage point here is Orient.

The model we form of the world guides our decisions, both conscious and unconscious. Once the pilot has a geometric plane of battle in mind, the decisions are obvious. Once you see the bottleneck in production, you can’t look away from it. When I have an idea what’s going on in my daughter’s mind, I can talk to her.

Our power to change our actions, our habits, and our impact on the world lies in Orient. When we direct our attention to finding new models of the world, whole new possibilities of action open to us.

Fighter pilots can see what is possible when they picture the battle in the best geometric plane. Production managers needs to look at the flow of work. In software, I look at the flow of data through services and functions — different from when I used to see in objects or think about spots in memory.

The power of breaking work into smaller chunks is the chance to re-Orient in between them. TDD gives us lots of little stable points to stop and think. Pairing lets one person think about where we are in the problem space while the other is busy acting. Mob programming gives us the chance to negotiate an orientation among the whole group.

That co-orientation is crucial to collaboration. With that, we can predict each other’s decisions and understand each other’s actions. If we have a shared model of the world and when we are going, plus trust in the competence of our team in their respective specialties, that’s when we can really fly.

(This post is based on a conversation with Zack Kanter.)

Implementing all the interfaces

Humans are magic because we are components of many systems at once. We don’t just build into systems one level higher, we participate in systems many levels higher and everywhere in between.

In code, a method while is part of a class which is part of a library which is part of a service which is part of a distributed system — there is a hierarchy, and each piece fits where it does.

An atom is part of one molecule, which combines into one protein which functions in one cell in one tissue in one organ, if it’s lucky to be part of something exciting like a person.

But as a person, I am an individual and a mother and a team member and an employee and a citizen (of town, state, country) and a human animal. I am myself, and I participate in systems from relationship to family to community to culture. We function at all these levels, and often they load us with conflicting goals.

Gregory Bateson (PDF) describes native Bali culture: each full citizen participates in the village council. Outside of village council meetings, they speak for themselves. In the council, the speak in the interests of I Desa (literally, Mr. Village).

Stewart Brand lists these levels of pace and size in a civilization:

  • Fashion/art (changes fastest, most experimental)
  • Commerce
  • Infrastructure
  • Governance
  • Culture
  • Nature (changes slowest, moderates everything else)

Each of these work at different timescales. Each of us participates in each of them.

We each look out for our own interests (what is the fashionable coding platform of the day) and our family and company’s economic interest (what can we deliver and charge for this quarter) and infrastructure (what will let us keep operating and delivering long-term) and so on.

Often these are in conflict. The interests of commerce can conflict with the interests of nature. My personal finances conflict with the city building infrastructure. My nation might be in opposition to the needs of the human race. Yet, my nation can’t continue to exist without the stability of our natural world. My job won’t exist without an economic system, which depends on stable governance.

If we were Java classes, we’d implement twenty different interfaces, none of them perfectly, all of them evolving at different rates, and we’re single-threaded with very long GC pauses.

Tough stuff, being human.

Rules in context: D&D edition

In Dungeons & Dragons (the tabletop game), there are universal laws. These are published in the Player’s Guide. They set parameters for the characters, like how powerful they should be relative to monsters. The Player’s Guide outlines weapons, combat procedures, and success rates. It describes spells, what they do and how long they last. What is a reasonable amount of gold to pay for a weapon, and how much learning (XP) comes from a fight.

The Players Guide does not tell you: everything else. What happens when a player attempts to save a drowning baby using a waffle?

The Player’s Guide represents the universal laws of D&D. The rules exist because they’ve been shown (over time, this is the 5th edition) to enable games that are fun.

Yet the prime directive of D&D is: what the DM says, goes. (The DM is the dungeon master, the person telling the story in collaboration with the players.) The DM can override the rules when necessary. More often, the DM makes up rules to suit the situation. The rulebooks do not cover everything the players might choose to do, that that’s both essential and by design.

In D&D, the DM sets the stage with a situation. Then the players respond, describing how the characters they control act in this situation. The DM determines what happens as a result of their actions.

In our game today, Tyler was DM. Tyler DMs by the “Rule of Cool”: “If it’s cool, let them do it. If it’s not cool, don’t make them do it.” One character, TDK Turtle, ran out of the inn with a waffle in hand. On his next turn, he tried to use the waffle to save a drowning baby.

Could that ever work? The DM decides. How unlikely is this? More unlikely than Turtle rolled. And yet Tyler came up with a consequence: Turtle threw the waffle in the river, our dog jumped in to eat the waffle, the baby grabbed onto the dog, and thus the dog saved the baby.

Every D&D campaign (series of games with the same DM and roughly the same players) has its own contextual rules. These build up over time. Our party has a dog because yesterday we rescued this pet from a Kuo-toa tribe that was trying to worship it as a Doge. (The Kuo-toa worship gods of random construction. Where by random I mean, DM’s choice. This DM chose Doge, because it advanced the plot.)

What works for a group of players, we stick with. What doesn’t, we leave behind. If it’s cool, do it. If not, don’t. Results drive future practices.

Our teams are like this. Humans work within universal laws of needing to eat and sleep and commute. Organizations impose constraints. Within these bounds, we come up with what works for us, what makes us laugh, and what helps us advance the plot of the system we are building.

Not every baby-saving-waffle-toss is the same. Not every party has this dog. Let teams build their own process, and don’t expect it to transfer. Do look for the wider rules that facilitate a productive game, and try those more broadly.

These are not the only options. Wineglass edition

Today I found myself in the kitchen, near the fridge with the wine (it’s an excellent Chardonnay from Sonoma, thanks XYZ Beverages in Tybee Island, you exceed my expectations although you don’t have a website to link to). My empty glass was out on the screened porch.

Do I go outside for the glass? Or take the wine bottle to the glass, and then return it to the fridge?

These are not the only options. I snag another wineglass from the cupboard, fill it with wine, and take that out to the porch.

Now I have two dirty wineglasses, but who cares? The dishwasher washes them all at the same rate.

This is garbage collection in action. The dishwasher acts as a garbage collector for dirty dishes. It adds the capability of “do not worry about how many dishes you dirty. They will all be cleaned for the same fixed cost that you have already incurred.”

This removes one consideration that I need to think about in my actions. I’m free to optimize for my higher-level objectives (“be on the porch, with wine in a glass”) while ignoring the accumulation of resources (dirty wineglasses). It takes some adjustment to benefit from this condition.

It takes some adjustment in a brain to move from scarcity (“Dishes are a resource with a cost”) to abundance (“dirty dishes meh, not a problem anymore”). Once adjusted, the options opened to me are widened, in a way that a clearly optimal path is opened.

Now pardon me while I finish this delicious glass of wine and fetch another, from the nice cold bottle still in the fridge.

Zooming in and out, for software and love

The most mentally straining part of programming (for me) is focusing down on the detail of a line of code while maintaining perspective on why we are doing this at all. When a particular implementation gets hard, should I keep going? back up a step and redesign? or back way up and solve the problem in a different way?

Understanding the full “why” of what I’m doing helps me make decisions from naming to error handling to library and tool integrations. But it’s hard. It takes time to shift my brain from the detail level to the business level and back. (This is one way pairing helps.)

That zooming in and out is tough, and it’s essential. This morning I learned that it is also essential in love. Maria Popova quotes poets and philosophers on how love requires understanding, then:

We might feel that such an understanding calls for crouching closer and closer to its subject, be it self or other, in order to examine it with narrow focus and shallow depth of field, but this is a misleading intuition — the understanding of love is an expansive understanding, requiring us to zoom out of our habitual solipsism so as to regard ourselves and the object of our love from a great distance against the backdrop of universal life.

Maria Popova, Brain Pickings

Abeba Birhane, cognitive scientist, points out that Western culture is great at picking things apart, breaking problems up to their smallest possible components. She quotes Prigogine and Stenges: “We are so good at it. So good, we often forget to put the pieces back together again.”

She also sees this problem in software. “We forgot why we are doing it, What does this little component have to do with the big picture?” (video)

Software, love, everywhere. Juarrero brings this together when she promotes hermeneutics as the way to understand complex systems. Hermeneutics means interpretation, finding meaning, especially of language. (Canonical example: Jews studying the torah, every word in excruciating detail, in the context of the person who wrote it, in the context of their place and time and relations.) Hermeneutics emphasizes zooming in and out from the specific words to the work as a whole, and then back. We can’t understand the details outside the broader purpose, and the purpose is revealed by all the details.

This is the approach that can get us good software. (Not just clean code, actual good software.) I recommend this 4-minute definition of hermeneutics; it’s super dense and taught me some things this morning. Who knows, it might help your love life too.

Knowledge resides in teams

The magic of a gelled team is that they know how to work together, and together, they know how to do particular work. The members don’t know how to work together; the team does.

These learnings don’t reside in the members individually, the learnings are in the interrelations.

A shared know-how is jointly constructed between the participants. This shared know-how does not amount to the sum of the individuals’ know-hows nor does it strictly “belong” to any of the participants…. It involves instead the practice of coordinating sensorimotor schemes together, navigating breakdowns, and it belongs to the system the participants bring forth together: the dyad, the group, the family, the community, and so on.

“Linguistic Bodies The Continuity between Life and Language” Ezequiel A. Di
Paolo, Elena Clare Cuffari, and Hanne De Jaegher, quoted by @theblub

If you’re a director who gets to decide which teams stay together and which break apart, you have a lot of power — and very little control. Power can bust up symmathesies, but not build them or repair them. Other levels of hierarchy can set up conditions for success, but teams grow from within.

Leaving a company is scary because we know how to be in that company. Our own knowing-how-to-be exists partly in our interrelations there. Finding a new job means discovering a new way, a new self, to be in the new place. With families, even more so – this is part of what makes divorce so scary. If you’re in an unhealthy system and can’t imagine anything else, this is normal.

When you do get to be part of a healthy team, or a healthy family, appreciate it. Cherish it and nourish it.