No Roadmaps for Worldbringers

Goals are for the smaller stuff,
specifics of this week, today
because we can’t see far enough
to guess what new surprises lay
beyond that. Top this nearest hill
and pause to find the next clear way.
A mountain in the distance still
calls to us, and yet we stay
grounded in the valley here
and set a goal not far away.

Tomorrow, top this next near hill —
the valley grows a different shape.
This landscape wasn’t formed until
we moved here. We create the place
we stand in; then we look for how
the world responds. This gives us play
to find where rock and rain allow
a level spot, where we can say:
We’re closer to that distant peak,
a taller state toward which we aim.
Because from there we just might reach
the star! And then the world will change
again.

Landing Zones, Long-term Desires, and Impossible Dreams

How do we get from here to where we want to be? Hint: don’t draw a roadmap. The road we’ll travel in six months doesn’t exist yet.

Landing zones

Landing zone: “an improvement that would feel like an accomplishment, as well as a pause point to catch breath, reassess, and plan how to achieve the next better thing.”

Esther Derby, 7 Rules of Positive, Productive Change

Big changes come from small changes. Good thing, since small changes are the ones we can make.

We create the path to where we will be one setpping-stone at a time. The smaller the step, the better.

Find the smallest step you can take that puts you one detectable bit closer to where you want to be — or, to where you have more information to know where you are and how to get where you need to go. Make it specific, so you know when you’re there and it’s time to look around.

Does your site need a redesign? First, improve the help text on a confusing button. Or, add events that show where people are using the interface in an unexpected way.

Small progress is the best progress, and information is also progress.

Long-term desires

Landing zones link “near-neighbor states” with long-term desires.

Esther Derby, 7 Rules of Positive, Productive Change

The bigger states we’re trying to achieve can be less specific than the landing zones we use day-to-day. They can tell us when it’s good enough, and it’s time reprioritize.

Maybe: 80% of customers who start make it all the way through enrollment.

We may still reprioritize before getting all the way up this mountain, but this gives us a direction to look in for day and week goals.

Esther suggests making a horizon map (sadly, this term does not google well) to get from long-term desires to landing zones. Map backwards from the desire by asking, “What conditions would have to exist for this to be true?” until you get to the current state. Then move forward one step, one landing zone, and check in.

Impossible Dream

You might achieve a long-term desire. Then what? You pick a new one, based on your organization’s purpose.

Everyone needs a direction to look in for the next mountain to climb. This purpose should not be achievable, because then your organization should close! We need a Quest, an Impossible Dream.

This is:

  • build better airplanes than the world has ever seen
  • help everyone in the world start their own business
  • advance systems thinking in software until the whole world works better

(It reminds me of John Cutler‘s North Star, and long-term desires are themes in that framework.)

We need a star over the horizon to point the direction, a long-term desire as a big step in that direction, and many landing zones along the way to detect progress and assess the landscape.

Keep commitment-style goals down to the landing zone level, so that we can keep our heads up and pointed toward our star.

PowerShell equivalent of ack

Today I found a typo in some docs: “account” was misspelled as “accuont.” I want to find that in the repository. As a bash user, I tried

ack accuont

but of course that command is not found in PowerShell.

It took me a while to find the equivalent so here it is:

Select-String -Path .\**\*.md -Pattern accuont

You need to list the fileglob or else it’ll search binary files and mess up your screen.

Comma-separated lists work for both globs and patterns. Regex strings work in patterns.

Source: https://devblogs.microsoft.com/scripting/use-an-easy-powershell-command-to-search-files-for-information/

Flow of what?

We talk a lot about optimizing flow in our software teams. Flow of what?

Flow of value to customers? The software takes care of that. Every pageview, data update, or call to our service outputs some value.

The people on the team are busy changing the software, trying to increase the value it outputs. The team’s flow is changes to the flow of value.

Then in our retrospectives, in articles like this one, in books and podcasts and conferences, we try to increase and improve the changes. Our flow right now is changes to the flow of changes to the flow of value.

Yikes, very meta! Very powerful, and very tricky. No wonder we talk about it so much.

(This topic came up when I interviewed the authors of Team Topologies on ArrestedDevOps the other day. Listen to the podcast for more on how to organize teams, improve these flows, and deliver software.)

Not about me

In a single-player video game, I’m a kid again, because the story is all about me. I am the main character, and every other character exists for me. They’re all standing around waiting for me to react to them. Everything they do, they do it to make me feel something.

The decisions I make change the world, inside the game. The world grows as I explore it. Nothing is accomplished unless I personally accomplish it.

Children generally start with this perspective. Everyone else exists in order to take care of them. Growing up feels (in part) like a process of letting go of this.

When a person snaps at me, or cuts me off, or says something horribly offensive — it is not about me. They have their own concerns, and my feelings don’t register.

When I complete a task, or make something cool, or flub and do something seriously awkward — it doesn’t matter. It is not about me. Only the people closest to me notice, and they quickly forget.

When I have a useful idea and must get it into the world right away, and then my kids need something and a deadline rears up, the world is deprived of this crucial output! … nah. It is not about me. When the world is ready for an idea, that idea comes to many people. Someone else will get it out there; they probably already have.

Gerald and Dani Weinberg call it the Law of Twins: most of what we do has no lasting effect.

There is freedom in this. People aren’t waiting to see what I will do. The world goes on. It is not about me.

What others do and say is about them and their context, not about me. The world exists in vast complication whether I perceive it or not. Time and culture move forward with my little contributions or without them.

The exception is: my family and close relationships, my team and collaborators. These people see and hear me, they feel and act partly in response to what I do. And it is my responsibility to see and react to them. I make a difference in the day my children have, in what my team accomplishes.

Cherish these people, and put care into your interactions with them. The rest of the world, meh — it is not about you. It’s okay if you never change it. It’s okay if you do; don’t fear affecting people. You mostly won’t, and if you do, the world was ready for it.

Then if you want to feel important, like the world revolves around you again, play video games.

Smaller pieces, lower pain

Part of XP is breaking work down into the smallest possible pieces. Kent Beck teaches us to teeny tiny changes, changes so small that you don’t mind starting over when you get things wrong. Llewellyn Falco breaks work down into bits so tiny that most of them provable refactorings, minute changes like putting “if (true){}” around some code; adding an empty else statement; or calling a function that does nothing.

When changing a complex system, it helps to make each change as simple as possible.

When our limitation is cognitive load, the difficulty of the task is not the sum of the difficulty of the steps. It is the maximum difficulty of any one step.

At home, when I get stressed out, when the kids are talking to me and I’m trying to get ready to go and the kitchen needs clean and what is sticky thing I just stepped on — I catch myself, and start breaking down my work into the smallest steps possible. One tiny thing at a time.

Put down what I’m holding. Listen to the child. Ask the child to wait. Fetch a washcloth. Get it wet. Wipe the sticky floor. Put the washcloth in the bin. Put one mug in the dishwasher, and call the kitchen “cleaner.” One at a time, put away the things I was holding. Walk past the closet where my coat is, put on my shoes. Now get the coat. Put it on. Now get a hat. Put it on. Now get car keys. Now put the keys down. Put on gloves. Pick up the car keys. Ask the child to repeat the question.

This might take more clock time than if I try to answer the child and put away the stuff I’m holding and pick up the stuff I need while optimizing the route to avoid doubling back in the hallway. Maybe.

The tiny steps lower my cognitive load. This leaves me enough attention to hear the child’s question. It lets me handle the hardest single step (leaving the rest of the kitchen alone) without bitching.

Even easy things get hard when we lump them together. Add stress, and cognitive load is exceeded, leading to more stress, leading to things getting harder. Soon I’m yelling at the children, dropping a mug, and leaving without a hat.

Our limitation is not what we can do. It’s how much we can hold in our heads. So don’t push it!

In programming, it’s dangerous to work near your working memory threshold. You get more mistakes and more complicated code. In life, it’s stressful to optimize for fewer steps or fewer seconds on the clock. Do that when you’re bored; keep yourself entertained by straining your working memory. Only at home, please, not at work.

Great software teaches

Great software solves a problem that you have — plus problems you didn’t know you had.

Here’s an example: today on Twitter, a friend let me know about a broken link to one of my old posts:

The broken link lives in someone else’s blog post, so I can’t update the source. It looks like the link has been broken since I migrated from Blogger to WordPress.com some months ago. Darn!

Ideally, the old link would redirect to the correct one, the one Gary found after a bunch of looking. This would fix the internet (just a tiny bit).

How hard is it to make that work?

Look at this beautiful plugin that appears right at the top when I search for “redirect”:

Redirection plugin. It has a million zillion installations

Perfect. I installed it (thanks to $23/month to WordPress.com for my business site) and entered the two URLs and poof, the redirect worked. That took under 10 minutes.

But wait! There’s more!

During the installation it asked me whether I wanted logs of 301s (requests to my site that got redirected to the right place) and 404s (requests to my site for a page that is not found). Yeah, sure.

After entering the one redirect I knew about, I saw it work in the log of 301s. Then I clicked on the 404 report, and in that couple minutes it had already noticed two more broken requests!

part of the 404 report. It shows the link I just fixed plus two others I did not expect

So I fixed those too! Hovering over the link in the report even gives an “Add redirect” option that populates half of it for me. Amazing.

This Redirection plugin is great software. It worked to solve my specific problem. It teaches me more about the wider problems I’m having, and helps me solve those too. Brilliant.

Understanding, inside and out

Every company, every team is its own system and works in its own ways. There are universal abstractions, but these are only useful when someone can translate them into the particular context of one company or team.

Corporate anthropologists do this. First, they adopt the role of “participant observer.” They get deep into the context of teams and workers at many levels of an organization. They stand on the factory floor, they ride along, they share kitchens and coffee breaks. All the time with “the ever-present notebook in your pocket, jotting down observations.”[1]

They learn the inside perspective. To explain how the system works in the language of the people inside it, the way they experience it.

Then, the anthropologist considers what they saw from the outside perspective, in the frames of various theories. How do these particulars fit into universals of leadership, organization, and work?

This combination of inside and outside perspective provides insights invisible from either one. They might see where the intentions of leaders are lost in communication. They might see that work gets done only by circumventing certain rules.

I am not an anthropologist, but I want this kind of understanding. I want to see the workings of my team both from inside and from outside, to recognize what is particular and what is universal. All the time while doing work.

I am an observing participant.

Developing software with a notebook in my back pocket, I notice how my team gets work done, what rules they circumvent and what unstated conventions they enforce. I notice when I feel surprise or frustration and when others do — clues to deviance from unspoken patterns.

As a member of the team, the inside perspective is natural. I take conscious steps to learn the outside perspective.

I talk with other people at meetups and in Slack communities. Read books and dig into conferences. Seek frameworks and theories of work in online materials and workshops.

Combining this outside view with my natural inside view lets me think about the wider purpose of our work, identify paths that can help us reach the goal more usefully, and flex when the wider system’s needs change.

Do the work, and while watching work. Seek outside perspective. Afterward, reflect. Be an observing participant.

[1] source: Danielle Brown and Jitske Kramer, The Corporate Tribe. Technically they talk about the two perspectives as emic (inside) and etic (outside).

illusions of commonsense perception

Learning is a struggle against “the illusions of commonsense perception” (Maria Popova). When it was obvious that the Earth stood still and the Moon moved, Kepler wrote a novel about traveling to the moon and meeting a civilization who believed, based on their commonsense perception, that the Moon stood still and the Earth moved. If a person can imagine the perspective of a moon-based being, maybe they can see that their belief is tied to their Earthbound context.

The phrase “That’s common sense,” like “That’s obvious,” means “I believe this, and the people I trust all believe this, and I can’t explain it.”

Many people in America have a commonsense notion: “Men have penises and women have vaginas.” It’s all they’ve personally seen. Can you imagine someone with a different perspective? an intersex person, even if you don’t know that you’ve met one?

A woman shaving in dramatic light

This frame is self-fulfilling. Any one who is intersex ain’t gonna tell you about it, when you are sure that’s not a thing. Common sense shapes our perspective of the world in multiple ways: in what we can perceive, and in what people show us.

Especially if you have power! Then people extra show you what you want to see and nothing else. Power inhibits perception.

Common sense is contextual. In the Midwest, obviously everybody drives. (Scientifically, it’s incredibly dangerous.) In embedded device drivers, obviously you test your software thoroughly, and statically typed languages are superior. In web apps for ad campaigns, obviously that’s all a waste of time. Can we imagine situations where people have different perspectives?

If you imagine someone with a different perspective, then you can gain insight into your own perspective. You might get more accurate theories — be able to predict eclipses, which a lifetime of seeing the moon come up at night can’t prepare you for. You might see that your perspective isn’t true everywhere, and that you could move — maybe not to the moon, but to a city where there’s a community of queer or genderfluid people (and maybe even public transport). Or to a team that treats testing differently, where you can expand your experience.

But this kind of imagination takes work. Was it useful to the average human in 1600 to know that the Earth revolves around the sun? Heck, is it to the average person today? It isn’t directly relevant to my daily life, but I believe it by default, because everyone around me does. Most of the commonsense beliefs we grow up with aren’t worth questioning.

It takes effort, research, emotional energy and brainspace to adopt new frames. We can’t all do it for everything. Some of us learn that the gender binary is nonsense. Some of us learn many programming languages. Some of us study astronomy.

When you meet a person who still holds an outdated notion, it doesn’t mean they’re an idiot. It means they haven’t taken the effort to break this one yet. We can’t all understand everything. And most of the time we don’t need to. It’s a bonus when someone does break out of the default beliefs.

When you do gain new understanding and alter the beliefs you started with, it stays with you forever. Wisdom comes with age, or with accumulation of shattered assumptions.

Kepler understood this, and he worked to make it easier for people to understand that maybe Earth isn’t the only place in the universe, and therefore not the center of the universe. Thank you to people who share their stories, lowering the effort it takes me to realize that a belief that’s been good enough for this long, is not good enough for everyone.

Capturing the World in Software

TL;DR – we can get a complete, consistent model of a small piece of the world using Event Sourcing. This is powerful but expensive.

Today on twitter, Jimmy Bogard on the tradeoffs of Event Sourcing:

If event sourcing is not scalable, faster, or simpler, why use it?

Event Sourcing gives you a complete, consistent model of the slice of the world modeled by your software. That’s pretty attractive.

We want to model the real world in software.

You can think about the present world as a sum of everything that happened before. Looking around my room, I can say that my bookshelf is the sum of various purchases, some moving around, a set of decisions about what to read and what to keep.

my bookshelf has philosophy, math, visualization, and a hippo

I can think of myself as the sum of everything that has happened, plus the stories I told myself about that. My next action is an outcome of this, plus my present surroundings, plus incoming events. That action itself is an event in the world.

In life, in biology, we don’t get to see all these inputs. We don’t get to change the response algorithm and try again. But in software, we can!

Of course we want perfect modeling and traceability of decisions! This way we can always answer “why,” and we can improve our understanding and decisionmaking strategies as we learn.

This is what Event Sourcing offers.

We want our model to be complete and consistent.

It’s impossible to model the entire world. Completeness and consistency are in conflict, sadly. Still, if we limit “complete” to a business domain, and to the boundaries of our company, this is possible. Theoretically.

Event Sourcing offers a way to do that.

In event sourcing, every piece of input is an event. Someone requests a counseling appointment, event. Provider signs up for available hours, event. Appointment scheduled, event. Customer notified, event. Customer shows up, event. Session report filed, event.

We can sum past events to get the current state

Skim the timeline of all events for the relevant ones. Sum these up (there are other definitions of “sum” besides adding numbers). From this we calculate the state of the world.

From appointment-was-scheduled events, we construct a provider’s calendar for the day.

At the end of the month, we construct reports on customers served and provider utilization. Based on that, we might seek more providers or have a talk with the less active ones. Headquarters ranks the performance of our office compared with others.

We need to allow corrections

To accurately model the real world, we need to allow for all the stuff that happens in the real world.

Appointments are cancelled. Customers don’t show up. Session reports are filed late. (“Where’s that session report from last week?” “Oh right, they were too late, because the gate to the parking lot malfunctioned. Don’t charge them for it.”)

Data is late or lost. If you insist that this doesn’t happen (“Every provider must enter the session reports by the end of the day”) then your model is blind to reality. The weather turns bad, people go home. There’s a bomb threat, or an active shooter. Reality intrudes.

Events outside your careful model will happen. Accommodate corrections, incorporate events that arrive late, accept partial data. The more of reality you allow into your model, the more accurate it can be.

We can evaluate past decisions based on the information available at the time

When data arrives late, reports change after they are printed. An event sourced system handles this.

As new data comes in about past days, it gets summed in with the data about those days. Reports get more accurate.

A friend of mine works at a counseling center, and he gets calls from headquarters like “Why is your utilization so low for December?” and he’s like “What? It was fine” and then he runs the report again and sure enough, it’s different. After he ran the report, more data about December came in, and now the totals are different. He can’t reproduce the reports he saw, which makes it hard to explain his actions to HQ.

If their software used event sourcing, he could say, “Please run the report as of January 2, and you’ll see why I didn’t take any action.”

Each event records a received timestamp, for when we learned about it, and an effective timestamp, for the real-world happening it represents. Then the software can sum only the events received before January 2 to reproduce the report as it was seen that day.

We can re-evaluate the world with new logic

Not only can an event-sourced system reproduce the same report as on an earlier day, we can ask: what if we changed the report logic? Then what would it look like?

Maybe we want to report unreported appointments as “possibly cancelled” to reflect uncertainty. We can run the new logic against the same events and compare it to the old results.

This means we can run tests against the event stream and detect behavior changes.

We need to record externally-visible decisions for consistency

When we change the software, we endanger consistency.

If we update the report logic in February, then when HQ runs the report “as of January 2” they’ll see something different than my friend saw when he ran it on that date. For consistency, both the data and code need to match what existed on January 2.

Or, we can model the report itself as an event. “On January 2, I said this about December.” Then we can incorporate that into the reporting logic.

Anything our system does that is visible to the outside world is itself an event, because it changes the way external people and software act. To reproduce our behavior consistently, our system can either record its own behavior, or retain all the data and the code that went into choosing it.

So far, this is nice and deterministic. But the real world isn’t.

Reproducing behavior is possible in an event-sourced system, if that behavior is deterministic. In human behavior, we don’t get that luxury. Our choices come from many influences, some of them contradictory. One tweet inspired me to write this article. Thousands of other tweets distract me from it.

Conflicting information comes in from real life.

Event sourcing gets tricky when the real world we are modeling is inconsistent, according to the events that come in.

Now say we’re a shipping company. We model the movement of goods in containers as they move across the world. It is an event when a container is loaded on a ship, and an event when it is unloaded. An event when a ship’s itinerary is scheduled, and when it arrives at each port.

One event says that container 1418 was loaded onto the vessel Enceladus in Auckland. Another event says that Enceladus is scheduled for its next stop in Beijing. Another event says that container 1418 was unloaded in San Francisco. Another says that container 1418 was emptied in Beijing. Which do you believe?

This example comes from a real story. Weird things happen. Does your system let people report reality? Is there a fallback for “Ask a person to go look for that container. Is it really 1418?”

Decisions made in ambiguity are events

Whatever decision the system makes, it needs to record that as an event. Perhaps that shows up as a footnote in reports about Enceladus, Beijing, and San Francisco. Does anybody hear about it in Auckland?

We can see the provenance of each report and decision

If some report comes out uneven, and that feeds back to the development team as a bug, then event sourcing gives us excellent tools for tracking it down.

Each “I made this decision” or “I produced this report” event can record the set of events that were input, and the version of code that ran to produce the output. You can have complete provenance.

This kind of software is accountable. It can tell the story of its decisions, what it did and why. What its world was like at that time.

This is a beautiful property. With full provenance, we can understand what happened. We can tell the story to each other. With replayability, we can change the code and see whether we can improve it for next time.

Recording everything gets ridiculous

Yet, data about provenance gets big very quickly. Each report consumed thousands of events. Each decision that was based on a current-state sum of events now has a dependency on all of those past events, plus the code that defines the current state, plus all the other states it took input from, plus their code and set of events.

Meanwhile some of those events are old, and no longer fit the format expected by the latest code. Meanwhile, we’re still ignoring everything that happened outside the system, so we’re completely blind to a lot of causality. “A person clicked this button.” Why? What information did they see on the screen as input to their decision to click “Container 1418 is in San Francisco”?

In real life, most information is lost. History will never be fully written; the writing is itself history. We’re always generating new actions. The system could theoretically report on all the reports it has reported. It never ends.

Completeness is limited to very small systems. Be careful where you invest this effort. Consciously select the boundaries, outside of which you don’t know what happened. You don’t know what really happened in the shipyard, or in a person’s head, or in the software that another company runs. The slice of the world we see is tiny.

Provenance is precious but difficult. Then again, it is at least as hard to do well in designs other than event sourcing. The painful realities that make event sourcing are painful in other models, too.

There are reasons we don’t model the whole world.

Event sourcing makes a best effort to model the world in its fullness. We try to remember everything significant that happens, sum that up into our own current-state world in the software, make decisions and act.

But events come in out of order. Events are lost. Events contradict each other. Events have partial data, or old data formats. Logic changes. We can’t remember everything.

Sometimes it pays to think about what you would do in an event-sourced system, and then implement just enough of that. Keep copies of produced reports, so that people can retrieve them without re-generating them. Record difficult decisions in a place that lives longer than logs.

Event sourcing is powerful. But it is not easy. Expect to think really hard about edge cases you didn’t want to handle. Expect to deal with storage and speed and up-to-dateness tradeoffs. Allow a human to enter corrections, because the real world will always surprise you.

In the real world, we don’t have all the information, and that’s OK. We can’t model everything in our heads, because our heads are inside everything. This keeps it interesting.