Step 1: On one team, put the people with the knowledge and control necessary to change the software, see the results, change it, see the results.
Step 2: Use automation to take extrinsic cognitive load off this team, so that it needs fewer people.
That’s it, that’s DevOps.
Step 1 describes the cultural change that leads to flow. Delivering change requires no handoffs or approvals from outside the team; the impact of change flows back to the team. Act and learn.
Step 2 is where tools come in. If all you do is improve your tooling, well, that helps a little, but it doesn’t get you the qualitative change in flow. That comes from Step 1. The serious value of automation is that it enables Step 1, a single team with all the relevant knowledge.
Our job as developers is making decisions. DevOps gets us the knowledge we need to make good decisions, the authority to implement them, and the feedback to make better ones in the future.
Superlatives are dangerous. They pressure you to line up the alternatives in a row, judge them all, make the perfect decision.
This implies that databases can be compared based on internal qualities. Performance, reliability, scalability, integrity, what else is there?
We know better than that. We recognize that relational databases, graph databases, document stores etc serve different use cases.
“What are you going to do with it?”
This is progress; we are now considering the larger problem. Can we define the best database per use case?
In 7 Rules of Effective Change, Esther Derby recommends balance among (1) inner needs and capabilities, (2) the needs and capabilities of those near you, and (3) the larger context and problem. (This is called “congruence.”)
“What is the best?” considers only (1) internal qualities. Asking “What are you doing with it?” adds (3) the problem we’re solving. What about (2) the people and systems nearby?
Maybe you want to store JSON blobs and the high-scale solution is Mongo. But is that the “best” for your situation?
does anyone on the team know how to use and administer it?
are there good drivers and client libraries for your language?
does it run well in the production environment you already run?
do the frameworks you use integrate with it?
do you have diagnostic tools and utilities for it?
Maybe you’ve never used Mongo before. Maybe you already know PostgreSQL, and it already runs at your company. Can you store your JSON blobs there instead? Maybe we don’t need anything “better.”
The people we are and systems we have matter. If a component works smoothly with them, and is good enough to do the task at hand, great. What works for you is better than anyone else’s “best.”
Today my daughter overslept and I had to take her to school. Usually when that happens, I get all grouchy and resentful. Dammit, I thought I’d get a nice quiet coffee but NO, I’m scraping ice off the car.
Today I didn’t mind. It was fine. It’s the first day back after winter break, so I expected her to oversleep. I expected to take her to school.
I expected to scrape ice off the car… OK no, I forgot about that part and did get a little grouchy.
Our feelings surface from the difference between expectations and reality. Not from reality by itself.
In our career, people may ask us what we want in our next role. “Where do you want to be in five years?” Answering this question hurts in several ways.
The more specific our vision of our future selves, the more we set ourselves up for disappointment.
We hide from ourselves all the other possibilities that we didn’t know about.
We tie our future self to the imagined desires of our present self.
In the next five years, I will gain new interests, new friends, and new ideas. Let those, along with the new situations I find myself in, guide my future steps.
Expectations are dangerous. We need some of them, especially in the near future, to take action. The less specific they can be, the more potential remains open to us, and the happier we can be.
Tomorrow, I will have a quiet coffee or take my daughter to school; I don’t have to know which until it happens. Five years from now, I have no idea what I’ll be doing — probably something cooler than my present self can imagine.
It’s difficult for an executive to criticize a budget when most line items are for mysterious high technology activities. It’s easier to tackle the more understandable portions, like postage, janitorial services, and consulting.
We want to help. We want to do stuff. We look for matches between what needs done and what we know how to do, and then we do it.
But does that always help?
Today in his newsletter, Marcus Blankenship told the story of the three bricklayers. What are you doing? “Laying bricks.” “Making a wall.” “Building a cathedral.”
We want to do something. We want to lay some bricks. As programmers, we want to write some tests, make a class, throw out an API.
After we break down a feature implementation into tasks, it is tempting to get started on the parts we know how to do. Knock those out, advance that progress bar. Make the wall taller.
Those are the least helpful parts to start with! In software development, our job is making decisions. What we need most is knowledge.
The pieces of the task most amorphous, kinda vague, the ones our brains want to slide past with hand-waves — those are the tasks that will give us more than apparent progress.
Integrate with that new service. Get authorization working. Pick the database and get familiar with it.
Digging into the uncertain tasks gives us information. We will learn how the API needs to be different than we thought. The data we didn’t know we needed, the unhappy-paths we didn’t know we needed to pave.
Lay the bricks slowly. Consider, what is going to hold this wall up? and what will this wall hold up?
In the opening quote, an executive tries to manage costs. They are drawn to the little nitpicky items that look approachable. But the easy ones are already pretty good! The giant Megatechnology items with big dollars next to them, these provoke handwaves. What might the executive gain by digging in and learning more about these? Maybe not cost-cutting, but definitely better decision making.
Today I brought up a load of laundry. When doing chores, I practice keeping WIP (work in progress) to a minimum. Finish each thing, then do another one. This is good training for code.
For instance, on the way up the stairs with the basket, I saw a tumbleweed of cat hair. I didn’t pick it up. Right now I’m doing laundry.
I put the basket on the bed, pulled out a pair of pants, folded it, and then stopped.
Do I put the pants on the bed and fold the rest? Or do I put the pants away right now, then start the next piece of clothing?
It depends which one thing I’m doing: folding this load of laundry? or putting a piece of clothing in its place?
It’s like in software development. Do we slice work by functionality or by layer?
Feature slicing, where we do all components (front end, back end, database, etc) of one small change and release that before moving on: this is like folding the pants and putting them away before picking up another item.
Layered work, where we first make the whole database schema, then develop the back end, then create the front end: this is like folding all the clothes and then putting them all away.
Pants on the bed are WIP. When clothes are on the bed, the cat sits on them and I can’t put them away. Then when I want to nap, my bed still has clothes on it. WIP is liability, not value. I can’t access my bed, and no one has clean pants.
Yet, folding the laundry and then putting it away is more efficient. I might fold three pairs of pants, and then put them away all at once. Four towels, one trip to the bathroom closet. The process as a whole is faster and smoother (excluding the cats).
Is layered work more efficient in software? NO. It always takes far longer, with worse results. A lot of rework happens, and then we settle for something that isn’t super slick.
Why is laundry different?
Because I’ve folded and put away this same pair of pants many times before. On the same shelf. Household chores are rarely a discovery process.
If I hadn’t done this before, then I might fold all of Evelyn’s pants in fourths. That is standard practice, and my pants fit nicely in my cabinet that way. When I go to put Evelyn’s pants away, I’d find that her shelf is deeper. It’s just right for pants folded in thirds. Folded in fourths, they don’t all fit; I run out of height.
Now it’s time for rework: fold all her pants again, in thirds this time.
With feature slicing, I would fold one pair of pants in fourths, put it on the shelf, notice that it doesn’t fit well, refold it in thirds, and find that it fits perfectly. Every subsequent pair of her pants, I’d fold in thirds to begin with.
Completing a thin feature slice brings learning back to all future feature slices.
For repetitive chores, we can choose efficiency. For new work, such as all software development, aim for learning. That will make us fast.
Games aren’t much “fun” when rules, rather than relationships, dominate the activity, when there is no attention to “flow,” “fairness,” “respect” and “nice.”
Dr. Linda Hughes, “Beyond the Rules of the Game: Why Are Rooie Rules Nice?”
At a past job, we played Hearts every day at lunch. Out of a core group of 6-8, there were always at least four to participate. I worked there for about five years; we played over a thousand games together. We wore out dozens of decks of cards.
On top of the core rules of Hearts, we accrued a whole culture. There was the “Slone shooter” (the worst possible score, named for a former member of the group). We said “Panda Panda” (winning a trick with the highest card) and “Where’s the Jerboa?” (the two of clubs comes out to start the game) — both originated in our favorite deck, which had a different animal on each card. Long after that deck retired, the cards retained their animal names.
We had rules of etiquette. The player in the lead was the target, and everyone else works together to damage their score. Everyone was expected to make a logically justifiable play, except when succumbing to other players chanting “Panda Panda!” to summon the Ace of Hearts.
I’ve never since had that much fun at cards. The rules of the game are only the beginning.
See, you can think about games, or you can observe gaming. There are the rules as written, and then there’s the experience of the players.
It’s the same with work: you can talk about work-as-imagined, or you can look at work-as-done.
Work-as-imagined is the official process. It is how you’re supposed to get your work done. Work-as-done is real life.
This is why tabletop board games are more fun than their electronic equivalents. Everyone sees how the rules work, because we execute them ourselves. House rules evolve. When there’s ambiguity, the group decides what’s fair. Lovely traditions grow, jokes get funnier with repetition, and the game becomes richer than its rules.
It’s the same at work. Post-its on the wall give shared physical context, and they’re more flexible than any ticket-tracking software. (Software usually limits reality to what was imagined by its developers.)
Each collaborating team eventually develops its own small culture. Vocabulary, jokes, etiquette. These exist on top of (sometimes in spite of) decreed processes of work.
These interactions make work “fun.” They care about “fairness,” “respect,” and “nice.” They also lead to “flow” — flow of work through the team. Communication is smooth, collaboration is joyful and productive. This is how we win.
Throwing food in the trash feels wasteful. Sometimes I feel compelled to eat the food instead. Then I feel lethargic, I gain weight, and everything I do is slower.
Sometimes waste isn’t a problem. The world is not better for that food passing through my digestive system. Sometimes it’s preventing waste that hurts us.
Inefficient code uses compute and costs money. Even when the response time and throughput don’t impact the user, that’s still wasteful, right?
Waste that speeds you up
Say we optimize all our code. But then it’s harder to read and to change. We slow ourselves down with our fear of waste.
Duplicate code takes time to write and test. Maybe many teams use some handy functions for formatting strings and manipulating lists. It’s a waste to maintain this more than once!
Say we put that in a shared utility library. Now every change I make impacts a bunch of other teams, and every change they make surprises me. To save a bit of typing, we’ve taken on coupling and mushed up the code ownership. Everything we do is slower.
Waste that slows you down
On the other hand, duplicated business logic means every future change takes more work and coordination. That is some waste that will bite you.
In Office Space, there’s that one person who takes the requirements from one floor to another. His salary is a waste. Much worse: he wants to preserve his meager responsibilities, so he’ll prevent the people who use the software from talking to the developers. Everything is slower forever.
When you spot a distasteful waste, ask: does this waste speed me up, or does this waste slows me down forever?
I can be wasteful, and it’s okay sometimes. I’ll waste compute for faster and safer code changes. I’ll spend time on utility code to skip tripping up other teams.
Some waste is just waste. It is time spent once, or money spent ongoing, and that’s it. Some waste makes us more effective, saves us cognitive load of having to think about another thing. Some inefficiencies let us be more effective.
Other waste is sticky. It drags you into spending more time in the future. It pulls you into handoffs and queues and coupling.
Fight the sticky waste that spirals into more drag in the future. Let the other waste be. Throw the extra food in the trash; your future self will move lightly for it.
Logic and culture have nothing to do with one another.
Jerry Weinberg, The Secrets of Consulting
A friend of mine works for a large government organization that runs a dam, extracting electricity from water and gravity.
They have internal software development, of course, and my friend described some impressive obstacles to change. Deploying a new lambda function is a challenge. My friend calls on contacts throughout the department, including friends from previous jobs that now also work there. They help each other through procedural barriers.
I said, wow. He said, yeah, we have a mantra: “We make power, not sense.”
Culture doesn’t make sense, to anyone from outside. Culture is common sense, to anyone embedded in it.
To understand and work with a large organization, let go of trying to make sense of it. Observe it and see what’s there. After that, logic might help in finding ways to work skillfully inside it, maybe even to change it.
Simon Wardley always says, map what is. Then think about moving it toward what you want.
Today before breakfast, we started the dishwasher. I think they might have been clean already, but when I saw Avdi loading all the dirty cups from the counters in, I didn’t complain.
Today making scrambled eggs, I didn’t have my favorite spatula. It was in the dishwasher. Dang it! This is a danger of running the dishwasher during the day: some dishes are unavailable.
Then after breakfast, I couldn’t put the dirties in the dishwasher, because it was running. I had to leave them in the sink or wash them by hand.
My brother-in-law, Tyler, has a system for this. Every night he starts the dishwasher, and every morning he empties it. Not full? Oh well, start it anyway.
He always knows whether they’re dirty or clean, based on the time of day. He has access to all his favorite spatulas all day. He can load the dishes into the dishwasher as soon as they’re dirty. For added bonus: the dishwasher makes its noise while no one is in the kitchen. He can predict his usage of dishwasher pellets: one per day.
Why doesn’t everyone do this? Waste, I guess. The dishwasher uses water and electricity to wash a partial load of dishes. To avoid this waste, I maintain state: is it full yet? Are the dishes clean or dirty? Am I going to need this spatula before it’s done? Why are these dirty dishes in the sink?
What if the dishwasher is smart enough to use less water and effort when there are fewer dishes? The washing machine totally does this. What if we can take care of the waste by add complexity to the dishwasher, leaving my life less complex so I can concentrate on the scrambled eggs?
Maybe tracking all this makes me feel more in-control of the kitchen. Maybe it feels like part of the core work of housekeeping. But it ain’t like anybody’s standing there admiring it. The value I provided this morning wasn’t dishwasher management; it was scrambled eggs.
When programming, keeping track of memory usage used to be part of the necessary complexity. Now that we have efficient garbage collection, we don’t have to use brainspace on that. We aren’t less powerful programmers because we leave memory recovery to the garbage collector; we are more powerful because we can use our brains on the germane complexity of the business.
The system as a whole, with a smarter dishwasher or garbage collector, is more complex. But that’s fine, because there’s a boundary; someone else understands dishwashing or memory management. A lot of complexity in that component relieves me of a little complexity in kitchen management, and that means I can handle more complexity in my chosen domain. Or in programming, I can think more at a higher level because someone else is thinking hard at a lower level.
Starting tonight, I’m gonna let the dishwasher do a little extra work. Then I will concentrate on the work people care about, and make better scrambled eggs.
Most software gets harder to change as it ages. Making modern applications, it is not enough to write a system and put it out there. We need continual improvement and adaptation to the growing world.
How can we develop and operate increasingly useful software?
To answer this, I need a mental model of how software teams work.
My model is symmathesy: a learning system of learning parts. A great software team is made of people, software, and tools. The people are always learning, the software is changing, the tools are improving.
With this view, let’s look at the question.
The external output of the team is: useful software, running in production. To make it increasingly useful, grow the symmathesy. Growth in a system is defined as an increase in flow. In a symmathesy, that means growth is an increase in learning.
The people learn about the software by operating it, mediated by tools. Then we teach the software and tools by developing. Unless development & operations are in the same team, learning is blocked.
To increase usefulness, the team needs to learn about use. We can get input from outside, and study events emitted by the running software, mediated by tools.
What we know about usefulness leads the team to design change, and move the system to the next version of itself. That next version may be more useful to the outside, or it may be better at learning and teaching, the better to increase knowledge inside the team.
Someone today told me, “All great things in life come from compounding interest” — from feedback loops. (Compound interest is the simplest familiar example of a feedback loop.)
In a great software team, that “compound interest” is our relevant knowledge. Of the software, of its environment, of the people and systems who use it.
Maximize learning to increase the usefulness of software, at an accelerating pace.