Subtasks vs Capabilities

There exists a myth about software: that we can start with what we want it to do, break that into subtasks and subtasks of subtasks that we can each specify completely, implement all those subtasks, and then assemble them into something useful.

Harold Abelson explains it in this classic lecture (h/t @pesterhazy for the link). He also explains the problem, and a solution.

He draws the hierarchy of subtasks. If we learn something and need to make a change at a spot in the middle, the work below it is all invalidated. And changes to that specification may ripple out to the sides, ruining everything.

Look at all these elegant subtasks, suddenly rendered worthless

That hierarchy of subtasks seems rational, but it is not practical. Instead, Abelson points out, the way things really get done is in levels of capabilities: fleshed-out subdomains of the problem. If we want to build graphics, then we need a capability of making shapes, and for that we need a capability of locating points. Each of these has its own vocabulary and implementation. Each is coherent and broadly useful, specialized to a problem domain but not a single task.

Abelson is working in a Lisp, and Lisp makes it easy to write mini-languages to solve each problem. So his capabilities work out to levels of language. Build a language to describe points and locations, then geometry on top of that, then combinations on top of that.

When something changes, chances are you can express the new need at one of the higher levels without changing anything further down.

The capabilities (his “language levels”) use each other flexibly.

These are capabilities. We need the capability to describe points, then a capability to assemble them into geometries, and on that we can build a capability of combining those into useful images.

The capabilities depend on each other, and they have specifications, but each is broadly useful. They are built for a range of closely related purposes, not one specific place in the whole.

This myth also exists in our companies and societies.

A culture is like a big organization which assigns each of its members a place where he can work in the spirit if the whole…. In an age without culture… forces become fragmented and the power of an individual man is used up in overcoming opposing forces and frictional resistances.

Wittgenstein, Culture and Value

Here, Wittgenstein is very wrong. (To be fair, he didn’t publish this; the book is a posthumous collection of notes.) A culture certainly doesn’t break work into subtasks. But our companies do!

And it doesn’t work, either. You get rigidity, you get people held back by needing to go through others for specific subtasks. When infrastructure, purchasing, and marketing are elsewhere and we are required to delegate to them, this creates friction. The subtasks are never independent, so this doesn’t work even in theory.

Instead, create capabilities. For instance, to make useful software, we need infrastructure. Don’t create a subtask, assign it to a team, and make every software development team wait on them. Instead, create infrastructure as a service. Form a capability, not a fiefdom.

Build teams and software as capabilities, not subtasks.

“Software” isn’t a thing

You can’t make Software. There is no “software”–there is only “software that…” or “software for…”.

That’s like having a business around “writing.” Writing what?? Your writers are going to have to learn every topic they’re asked to write about.

That’s either going to take a long time, involve very close collaboration with someone who does know the topic, or be vapid and unimpressive.

If your core business is “software” — which of these is yours?

Certainty as blindness

How long before it dawns on us that the world we see no longer reflects the world we inhabit, that we are blind?

Blindsight, by Peter Watts

This quote is about the rare cases of people who are blind but insist they can see. And it is about all of us sometimes.

There was a model of the world, and we didn’t look outward at all; our conscious selves saw only the simulation in our heads, an interpretation of reality, endlessly refreshed by the senses.

painting by Cyndi Taylor

Until it isn’t refreshed anymore, until we retreat deep into our model and no longer perceive the differences in the physical world, or perceive them only as wrongness.

After decades in a waterfall world, where information officially passed through tangible documents, can a person see the casual flows of information in a product team working closely with a customer? Can they see the tests that happen in pairing with the developers? Can they see the frustrations cut short by pairing together?

They see wrongness, they see wasted time, they see information lost when it is not stored in a file cabinet. It takes releasing an old model to see what new things are happening.

Our model of how the world works is precious to us. It is precious because we have a place in it: we belong in that world, the world we grew into, the world that needs us.

Loosening that model hurts, it is disconnection, it pulls us away from what we know and love. But that world is gone. We can grow into the new one, if we can see it. To see it, we have to loosen our current model. It helps to seek being wrong.

Whether we want to change the world or find our place in it, it helps to see it. To see it as it is, right now, not as it was or as it ought to be.

That’s hard; by adulthood, we each have a model of the world that has worked for us. It is normal to go deeper into that model rather than shake out of it. Seeking out the places where our model is wrong can push us back into uncertainty, into vulnerability, into a breadth of possibility we haven’t owned since childhood.

We don’t grow up blind. We grow into blindness. We can grow out of it.

Complication for the win

“How can we make our programs simpler?”
I don’t want to make them simpler. I want to them make them complicated instead of complex.

Simplicity is not the goal.

Simpler implies they do less, they hold less rich domain knowledge. I want them to do a lot, to take work and thinking away from the user. Yet I still want them to be understandable by my team.

Like, say I’ve been to the grocery store and I’m stocking up on a lot of stuff. There’s a lot of stuff to bring in from the car.

How can I make this task simpler? Easy. Buy less stuff.

But I don’t want to buy less stuff. I’m gonna carry all this in. It feels like the simplest thing is to make one trip, load it all up in my arms. Head for the house and… shoot. Maybe I can kick the front door until someone opens it.

What seems “simple” might be the most complex.

Carrying all the groceries at once is a complex task. Every item I’m already carrying affects how I pick up the next one. Every extra pound feels significant, since I’m already encumbered. Every step is careful. And then I get to the kitchen and try to put it all down and crack the laundry detergent just came down hard on a carton of eggs.

Now that I’m older and wiser and have studied this phenomenon in software, I don’t try this anymore. I carry a few items in, then carry a few items in, etc etc until all the groceries are in the kitchen. This is more complicated because there are way more trips. It is less complex because each independent trip is simple.

Oodles of simple tasks are collectively complicated, but not complex.

I can carry two bags and a laundry detergent. One bag and the giant package of TP, easy. I can open the door for myself. I can put them safely on the counter.

I can take the detergent straight to the basement; every task doesn’t have to be the same as long as each one is simple.

As a bonus, my partner might start putting groceries away while I’m still carrying. The kids might join in and grab a few bags. When I’m carrying the whole mass they couldn’t grab one from me, because that would destabilize it all. But they can grab some from the car. Many simple tasks can be shared; one complex one cannot.

I’ll take complication over complexity any day.

In programs, it is tempting to see something as simpler if it has smaller parts. If one Order class has all the behaviors we need for an Order, that’s sounds simple. Then all those behaviors get intertwined and the Order class needs to change for many reasons and yet it’s scary to change because there are so many things we might break!

If we have a frobozz.purchasing.Order class that has behavior for building the order, and a frobozz.fulfillment.Order class with behavior around shipping the order, and a frobozz.billing.Order class with behavior around getting paid — gah! Three classes for one thing? Too complicated!

Yes, complicated. But far less complex.

Complication grows linearly, complexity… ugh.

twelve piles of groceries
twelve trips might be twelve times as complicated

Complicatedness can be measured (sort of) by a count of parts. Complicated systems can be broken down, broken down and each part understood alone, although that may take a lot of work because there are so many of them.

TP balanced on cases of beer held by arms barnacled in plastic bags — each part is leaning on another, interdependent.

Complexity is different. It exists in the interdependence between parts, which stops you from safely changing any of them independently. A complex system can only be broken down so far; you have to consider much of it in one lump. The most complex part determines the complexity of the system. It’s a maximum, not a sum.

Don’t be afraid of programs with a lot of parts. As long as those parts are independent — they communicate only with data, for instance, not interdependent database calls — then it doesn’t matter how many they are. Complication is not our enemy.

Aiming for “simplicity” by reducing complication leads us further into the morass.

Complexity bites us. Embrace complication, and keep your eggs safe.

How will you collaborate?

When there are decisions to be made, people have feelings about it. Some people want to make the decisions, others to influence it, others to hear about it.

Where does everyone in your group stand? Are you set up for conflict or for smooth collaboration?

For instance, say there’s a new product to start. We will need to decide how to market it, how to price it, and how to build it. Within building it, there will be all kinds of smaller decisions, from platform to architecture to every tiny piece of code.

Collaboration Contracts let us get explicit about how we will make decisions.

Doc Norton suggests: everyone pick your place in each decisionmaking. There are 6 options.

the six options as expressed by different colors of tooth people
which one are you?

Are you making the decision?

  1. Explainexplain person is orange“I will make this decision, and explain it to others.” No obligation to accept input.
  2. Consult: consult person is red“I will make this decision after consulting with others.” You can influence me, but I will decide.
  3. Agree: agree person is purple “We will make this decision by coming to agreement somehow.” Method of agreement to be determined later. You can influence us.

Otherwise, are you involved at all?

  1. Advise: advise person is green “You make the decision after I advise you.” I have thoughts; hear them and then I’ll respect the decision you make.
  2. Accept:accept person is blue“You make the decision. I’ll accept it.” Whatever it is, I’m in.
  3. Abstain:abstain person is grey “You do what you want. I abstain from this whole activity.” Why am I even in this meeting?

This lets us identify baked-in conflicts.

After everyone declares the role they expect to play in a decision, you can notice conflicts.

"explain" and "consult" and "agree" roles conflict with each other
“explain” and “consult” and “agree” conflict with each other.

If more than one person thinks they are making the decision, and they’re not all in the “agree” category, you have a problem.

"explain" and "advise" roles conflict with each other too

If someone thinks they’re making the decision unilaterally (the “explain” category), plus at least one person sees themself as “advise,” you have a problem.

Once you identify conflict, you can renegotiate. The person who thought they could make the decision without input can agree to ask others. The person who thought they would consult can seek agreement instead, or the people who wanted to agree can delegate the decision.

This model is useful at all levels of decisions. I find it especially interesting to apply it to low-level code decisions.

We can apply this model to collaboration on code.

Down at the nitty-gritty implementation level, someone is at the keyboard. Maybe its me. If I’m working alone, then I’m making the decisions. I am in the “explain” role.

After the code change is ready, what happens?

It’s OK to make unilateral decisions.

Maybe I push it to master, with an explanation of my decisions in a commit message. No conflict there.

Maybe I create a pull request, including an explanation of what I did and why. Then what?

If my colleage glances at it, sees that the build passed, and hits “merge,” maybe they’re taking the “accept” role and we’re fine.

There is conflict when we expect our decision to fly but it doesn’t.

But what if the reviewer has suggestions? Then they’re taking the “advise” role, and that is in conflict with my “explain” role. Looks like I actually have the “consult” role, except that I don’t get to consult until after I make the decision once and then I have to go make it again.

Pull request fight. Me: "Let's do this!" Them: "But what about..." Me: "Fine, they we'll..." Them: "That's not OK" Me (very sad): "Work with me here!"
the pull request slide, from decisionmaking to placation

And what if those comments aren’t suggestions, but blockers? Then we are both in the “agree” role. I thought it was my decision but it wasn’t; only I had to make the decision first and then someone else pops in and now we have to agree.

These are not smooth. I do not enjoy this.

We have ways of collaborating smoothly.

two "agree" people at one computer

If we were aiming for the “agree” role all along, why not start there? In pair programming, one of us is typing, but we make decisions together.

Or even better, mob programming: the person at the keyboard (the driver) is not allowed to make decisions, so every decision is spoken aloud. The driver is in the “accept” role.

mob programming with a navigator: an "accept" person with the keyboard; one "consult" person; several "advise" people

If your mob has a navigator, then each person takes a turn in “consult” and the rest are “advise.” This works.

mob programming: one "accept" person has the keyboard. Everyone else is "agree"

Without a navigator, everyone is “agree.” This works too.

Consider the design of your collaboration.

This model illuminates why pull requests hurt so much. They have conflict built in to their structure.

To make pull requests work, I have to lower my expectations.

I’ve learned to consider my pull requests to be suggestions. I’m proactively taking an “advise” role in a decision I hope some project committer will make. Success can look like: they implement the same functionality in a manner of their choosing. My PR functions as a very specific feature request. It doesn’t matter that my code didn’t make it into production, if my idea did.

The only exception to this is when I know the project maintainer (or am the project maintainer), and I come to agreement with them before submitting the PR. Then I expect it to be accepted, after some minor comments and changes are exchanged.

When you see conflict, consider your collective expectations.

Next time you feel resentment at someone else’s intrusion, or frustration at decisions made without you, try to picture the roles everyone thought they were playing. Are they compatible? Was conflict endemic to the structure of this interaction?

If so, don’t blame yourself, nor the other person.

Ask, can we be clear about our decisionmaking intentions? and then adjust them to remove conflict before it starts?

Be specific about your expectations. Set yourself up for smooth collaboration.

What makes a dependency dependable?

Lately I’ve been working in Node, and that means I have access to all┬áthe open-source modules on npm. It is easy to add a dependency on any of them. But is it a good idea?

Not every library is production-grade.

Not every library (or service) is something that I want my production code to depend on.

The first thing I look at when a module comes up in an npm search is: how many weekly downloads does it have? This is a cheat. “It’s production grade if other people use it in production” is a reasonable heuristic, but it doesn’t tell us how code becomes worthy of extensive re-use.

What does it take to make a production-grade library?

Some static qualities:

  • The code works as expected.
  • It is fast enough.
  • It has a well-designed API that lets me write readable code to call it.
  • The documentation is accurate and complete enough.

We look at:

  • What version is it on?
  • When was the last commit?
  • How many committers are there?
  • How many answered questions about it are on StackOverflow?

These touch on something deeper: how does the library change?

When I build my software on top of a library (or runtime, framework, toolchain etc), I want that library to be solid — and I want it to become more solid over time. I don’t want it to shift under my feet!

I want to know:

  • When there’s a security problem, a fix is released promptly.
  • Useful features are added (but not too quickly).
  • New features follow the pattern of existing APIs.
  • When I upgrade, my code will continue to work.

For bonus points:

  • Before I upgrade, I’ll still have access to the documentation for the version I use.
  • There is advance notice of upcoming features.
  • Someone from the core team responds to GitHub issues, StackOverflow questions, and tweets of despair.

I expect a production-grade library to practice responsible change.

Responsible change considers the larger world, all the people and software depending on the system that’s changing. It starts with backwards compatibility, regression testing, and careful API design. It continues with documentation, expectation-setting, and responsiveness to questions.

Very little of this happens in the code. Responsible change is a property of something larger. Whether in a company or a team of volunteers, a production-grade dependency is part of a symmathesy. Software resilience comes from people — people who have a mental model of how the software works. People who are actively involved in keeping it safe, and making it more appropriate for the shifting present.

That’s why we look for a team of committers, for recent attention, for a broad base of interest.

Most npm modules have a single author and haven’t changed in years. That’s fine! No open-source author is responsible to anyone for putting more work into that code than they darn well please. There are other ways to use this code than to depend on it.

If I can’t count on code to change responsibly, I don’t want it to change at all. I prefer to freeze the version of the library forever. If it needs to change, that’s my responsibility. I fork the library — or for similar effect, cut and paste the code into my project (with attribution).

This reduces the code that I have to write, but it does not reduce the code that my team is responsible for.

There are two kinds of open source: production-grade code with a community or company behind it, and code that I now own.

Next time you type “npm install,” make a guess as to which one you’re getting.

Landing Zones, Long-term Desires, and Impossible Dreams

How do we get from here to where we want to be? Hint: don’t draw a roadmap. The road we’ll travel in six months doesn’t exist yet.

Landing zones

Landing zone: “an improvement that would feel like an accomplishment, as well as a pause point to catch breath, reassess, and plan how to achieve the next better thing.”

Esther Derby, 7 Rules of Positive, Productive Change

Big changes come from small changes. Good thing, since small changes are the ones we can make.

We create the path to where we will be one setpping-stone at a time. The smaller the step, the better.

Find the smallest step you can take that puts you one detectable bit closer to where you want to be — or, to where you have more information to know where you are and how to get where you need to go. Make it specific, so you know when you’re there and it’s time to look around.

Does your site need a redesign? First, improve the help text on a confusing button. Or, add events that show where people are using the interface in an unexpected way.

Small progress is the best progress, and information is also progress.

Long-term desires

Landing zones link “near-neighbor states” with long-term desires.

Esther Derby, 7 Rules of Positive, Productive Change

The bigger states we’re trying to achieve can be less specific than the landing zones we use day-to-day. They can tell us when it’s good enough, and it’s time reprioritize.

Maybe: 80% of customers who start make it all the way through enrollment.

We may still reprioritize before getting all the way up this mountain, but this gives us a direction to look in for day and week goals.

Esther suggests making a horizon map (sadly, this term does not google well) to get from long-term desires to landing zones. Map backwards from the desire by asking, “What conditions would have to exist for this to be true?” until you get to the current state. Then move forward one step, one landing zone, and check in.

Impossible Dream

You might achieve a long-term desire. Then what? You pick a new one, based on your organization’s purpose.

Everyone needs a direction to look in for the next mountain to climb. This purpose should not be achievable, because then your organization should close! We need a Quest, an Impossible Dream.

This is:

  • build better airplanes than the world has ever seen
  • help everyone in the world start their own business
  • advance systems thinking in software until the whole world works better

(It reminds me of John Cutler‘s North Star, and long-term desires are themes in that framework.)

We need a star over the horizon to point the direction, a long-term desire as a big step in that direction, and many landing zones along the way to detect progress and assess the landscape.

Keep commitment-style goals down to the landing zone level, so that we can keep our heads up and pointed toward our star.

Not about me

In a single-player video game, I’m a kid again, because the story is all about me. I am the main character, and every other character exists for me. They’re all standing around waiting for me to react to them. Everything they do, they do it to make me feel something.

The decisions I make change the world, inside the game. The world grows as I explore it. Nothing is accomplished unless I personally accomplish it.

Children generally start with this perspective. Everyone else exists in order to take care of them. Growing up feels (in part) like a process of letting go of this.

When a person snaps at me, or cuts me off, or says something horribly offensive — it is not about me. They have their own concerns, and my feelings don’t register.

When I complete a task, or make something cool, or flub and do something seriously awkward — it doesn’t matter. It is not about me. Only the people closest to me notice, and they quickly forget.

When I have a useful idea and must get it into the world right away, and then my kids need something and a deadline rears up, the world is deprived of this crucial output! … nah. It is not about me. When the world is ready for an idea, that idea comes to many people. Someone else will get it out there; they probably already have.

Gerald and Dani Weinberg call it the Law of Twins: most of what we do has no lasting effect.

There is freedom in this. People aren’t waiting to see what I will do. The world goes on. It is not about me.

What others do and say is about them and their context, not about me. The world exists in vast complication whether I perceive it or not. Time and culture move forward with my little contributions or without them.

The exception is: my family and close relationships, my team and collaborators. These people see and hear me, they feel and act partly in response to what I do. And it is my responsibility to see and react to them. I make a difference in the day my children have, in what my team accomplishes.

Cherish these people, and put care into your interactions with them. The rest of the world, meh — it is not about you. It’s okay if you never change it. It’s okay if you do; don’t fear affecting people. You mostly won’t, and if you do, the world was ready for it.

Then if you want to feel important, like the world revolves around you again, play video games.

Smaller pieces, lower pain

Part of XP is breaking work down into the smallest possible pieces. Kent Beck teaches us to teeny tiny changes, changes so small that you don’t mind starting over when you get things wrong. Llewellyn Falco breaks work down into bits so tiny that most of them provable refactorings, minute changes like putting “if (true){}” around some code; adding an empty else statement; or calling a function that does nothing.

When changing a complex system, it helps to make each change as simple as possible.

When our limitation is cognitive load, the difficulty of the task is not the sum of the difficulty of the steps. It is the maximum difficulty of any one step.

At home, when I get stressed out, when the kids are talking to me and I’m trying to get ready to go and the kitchen needs clean and what is sticky thing I just stepped on — I catch myself, and start breaking down my work into the smallest steps possible. One tiny thing at a time.

Put down what I’m holding. Listen to the child. Ask the child to wait. Fetch a washcloth. Get it wet. Wipe the sticky floor. Put the washcloth in the bin. Put one mug in the dishwasher, and call the kitchen “cleaner.” One at a time, put away the things I was holding. Walk past the closet where my coat is, put on my shoes. Now get the coat. Put it on. Now get a hat. Put it on. Now get car keys. Now put the keys down. Put on gloves. Pick up the car keys. Ask the child to repeat the question.

This might take more clock time than if I try to answer the child and put away the stuff I’m holding and pick up the stuff I need while optimizing the route to avoid doubling back in the hallway. Maybe.

The tiny steps lower my cognitive load. This leaves me enough attention to hear the child’s question. It lets me handle the hardest single step (leaving the rest of the kitchen alone) without bitching.

Even easy things get hard when we lump them together. Add stress, and cognitive load is exceeded, leading to more stress, leading to things getting harder. Soon I’m yelling at the children, dropping a mug, and leaving without a hat.

Our limitation is not what we can do. It’s how much we can hold in our heads. So don’t push it!

In programming, it’s dangerous to work near your working memory threshold. You get more mistakes and more complicated code. In life, it’s stressful to optimize for fewer steps or fewer seconds on the clock. Do that when you’re bored; keep yourself entertained by straining your working memory. Only at home, please, not at work.

Great software teaches

Great software solves a problem that you have — plus problems you didn’t know you had.

Here’s an example: today on Twitter, a friend let me know about a broken link to one of my old posts:

The broken link lives in someone else’s blog post, so I can’t update the source. It looks like the link has been broken since I migrated from Blogger to some months ago. Darn!

Ideally, the old link would redirect to the correct one, the one Gary found after a bunch of looking. This would fix the internet (just a tiny bit).

How hard is it to make that work?

Look at this beautiful plugin that appears right at the top when I search for “redirect”:

Redirection plugin. It has a million zillion installations

Perfect. I installed it (thanks to $23/month to for my business site) and entered the two URLs and poof, the redirect worked. That took under 10 minutes.

But wait! There’s more!

During the installation it asked me whether I wanted logs of 301s (requests to my site that got redirected to the right place) and 404s (requests to my site for a page that is not found). Yeah, sure.

After entering the one redirect I knew about, I saw it work in the log of 301s. Then I clicked on the 404 report, and in that couple minutes it had already noticed two more broken requests!

part of the 404 report. It shows the link I just fixed plus two others I did not expect

So I fixed those too! Hovering over the link in the report even gives an “Add redirect” option that populates half of it for me. Amazing.

This Redirection plugin is great software. It worked to solve my specific problem. It teaches me more about the wider problems I’m having, and helps me solve those too. Brilliant.