When the db is the interface

There are two huge sources of inertia in software: data, and interfaces.

When two systems connect to the same database, you combine both. Ow.

When some other system is doing reporting on my database, I can’t change my table structure. That severely limits any internal reorganizations I might do. Especially when I don’t know how they use it, so I am afraid to touch it.

Then what if it’s a schemaless database? This means the schema resides in the application. When the schema resides in two applications at once, designing change is an exercise in hope.

Sometimes two systems access the same database because the database is the interface. We have an example of that at Atomist currently: one system (called squirrel, I don’t know why) populates a database, and another system (I call it org-viz) uses that data to make visualizations. How can that be okay?

Database as interface is not horrifying when:

  1. The database is not the system of record. We could repopulate all or part of it from elsewhere, so it’s somewhat disposable.
  2. The database has a schema. Interfaces must be well-defined.
  3. One system updates, the other reads.

In our case, we use PostgreSQL, so we have a schema. Some of our fields are JSON, which gives us extensibility, while the essential fields are guaranteed. The populating system (squirrel) operates from a queue, and we can replay the queue as needed, or play it into multiple databases. There are options for designing change.

Database as an interface is never going to be the cleanest decoupling, but it is not unreasonable when it is carefully designed and the teams don’t mind talking to each other. When the database is accidentally an interface, then you’re horked.

Feature interaction

Why is legacy software so much harder to work in?

Why does development velocity slow down as a system grows?

Lots of reasons, sure, but I suspect the biggest one is oft unspoken.

Every new feature comes with the invisible requirement: “and everything else still works.”

Every existing feature, and even past bugs, makes every new feature harder. Every user with expectations is a drag on change.

Feature interactions are hard! And they surprise you. My favorite examples come from The Sims release notes:

  • Fish are no longer duplicated in the fridge when moving homes.
  • Televisions no longer play video after they are burned or broken.
  • Sims will no longer walk on water to view paintings placed on swimming pool walls.
  • Pianists will no longer continue playing pianos that have been detonated.
  • Sims will no longer receive a wish to “Skinny Dip” with Mummies.
  • Sims who are on fire will no longer be forced to attend graduation before they can put themselves out.
  • Sims can no longer “Try for Baby” with the Grim Reaper.

Your feature interactions are probably not this much fun.

Software gets harder to change as it becomes part of a larger system, with more users and more uses. Development slows down because it’s useful, and we want it to stay useful. This also makes each change more valuable, because it’s helping more people in more ways.

Don’t be discouraged. Do be diligent, and be okay with being slower. This is the price of success.

Teams are like bread

Maybe when companies make you do “team-building” activities, what they’re looking for is a phase transition into a gelled team. Because it is a sudden, magical thing, right? When a group of people turns into a team.

Once you get there, to that feeling of team, it’s self-reinforcing. You trust each other, so y0u don’t take miscommunications personally; you work to restore communication, and so trust increases. You understand each other, so it’s easy to build further understanding. Working together gets smoother and smoother.

But how do you get there? pfft. How do you make sourdough bread? You grow it from sourdough starter, which you got from … someone else’s starter. Or from putting sugar water out on a windowsill and hoping some yeast lands in it. Seriously.

Will Larson suggests that when you have a gelled team, keep it. If you need to adjust how many people are helping with which pieces of software, then shift responsibilities from one team to another, not people.

When you have a gelled team, you can grow it gradually. Let the team reform with the new member incorporated before adding another.

Will suggests that when you want another team, gradually grow a gelled team up to 8-10 (max size) and then fork it into two teams of 4-5 (minimum size). It’s kind of like the sourdough starter: grow it, divide it, make the bread. Keep it alive the whole time.

If you have one team where the magic is flourishing, don’t kill it. Feed it, grow it, and let it be a source of further strong teams. No rushing.

Otherwise – if you take the group to paintball, or get them to mob program, or put them in a team room with sugar and water, maybe the yeast will blow in?

(if you want these little posts as I make them, plus a bit of extra context, sign up for my newsletter!)

Develop before define

First the loose thinking and the building up of a structure on unsound foundations and then the correction to stricter thinking and the substitutions a new underpinning beneath the already constructed mass.

Gregory Bateson on the advance of science. (From Steps to an Ecology of Mind)

This expresses a process I have observed in developers. We can develop something faster than we can define it.

That loose thinking includes the construction of loose code. We think with our fingers and eyes, keyboards and screens, editors and runtimes as well as with our brains. We try things, we draw them out or code them up. This eliminates a lot of impossible paths.

Then afterward, we shore up the useful ones. We put an API around it, error handling within, types throughout. We describe its interface and action in documentation.

Bateson grants permission to code loosely as an extension to thinking loosely, with the responsibility to return with rigor before we rope in other teams.

So do this, play in code the way we play in thought.

Then please realize that putting the foundations under it, defining the functionality so others can use it, is 10-100 times more time-consuming than your happy-path sketch.

When time gets tight, we make it tighter.

What do you think are the personality traits that contribute to being a good mathematician?

Flexibility. A willingness to change course when you see that things should go a different way. The ability to backtrack, and go forward and follow different paths and then come back to where you were. 

Dr Amie Wilkinson

This happens in code, too. When we explore a solution, we need to try one path, then come back and try another. This makes git incredibly valuable, with local branches and reverts.

I see Rod doing this often, when he works on exploring the design space of an API. (Rod is famous for creating the Spring framework for Java.)

If we are rushed, backing up can feel like wasting time. We push forward in directions that are slower. Worse, once someone else is using the API, it takes coordination to change it. That slows all of us down forever.

Yesterday Llewellyn Falco talked about how we tend to be prudent with money, leaving ourselves slack and valuing more-money-in-the-future more than less-money-now.

The multiplier of how much more money we require in the future is called the future discount. If you don’t think you’ll be alive in a month, or that you’ll really get any money at all then, your future discount is zero.

Under pressure, under scarcity, there is a psychological effect that reduces the future discount to zero. Saving money feels utterly pointless. Current needs are too pressing.

Llewellyn pointed out that while developers are prudent with money, they believe in the future, we often aren’t with time. We would not budget every dollar we have for a trip and leave no slack for surprises. But we do with sprints, and then we get under pressure and our future discount invisibly drops to zero and we can’t even think about future us or future whole-company who is stuck with the first design we thought to because backing up is just not an option.

Plow forward slower and slower, because we don’t believe in the future. Or step back and try a few things. Take a breath, take a walk, and maybe you’ll spot a smoother path.

Domain-specific laws

“there appear new laws and even new kinds of laws, which apply in the domain in question.”

David Bohm, quoted by Alicia Juarrero

He’s talking about the qualitative transformation that happens in a system when certain quantitative transition points are passed.

Qualitative transformation

I notice this when something that used to be a pain gets easier, sufficiently easier that I stop thinking about it and just use it. Like git log. There is such a thing as svn log but it’s so slow that I used it once ever in my years of svn. The crucial value in git log is that it’s so fast I can use it over and over again, each time tweaking the output.

  • git log
  • git log --oneline
  • git log --oneline | grep test
  • etc.

Now git log has way more functionality, because I can combine it with other shell commands, because it’s fast enough. This changes the system in more ways than “I use the commit log”: because I use the log, I make more commits with better messages. Now my system history is more informative than it used to be, all since the log command is faster.

The REPL has that effect in many languages. We try stuff all the time instead of thinking about it or looking it up, and as a result we learn faster, which changes the system.

Non-universal laws

I love the part about “laws, which apply in the domain in question.” There are laws of causality which are not universal, which apply only in specific contexts. The entire system history (including all its qualitative transformations) contribute to these contexts, so it’s very hard to generalize these laws even with conditions around them.

But can we study them? Can we observe the context-specific laws that apply on our own team, in our own symmathesy?

Can we each become scientists in the particular world we work in?

From Puzzles to Product

Note: this is a keynote from Velocity Conf 2019, San Jose. A text version of the material follows, and then references.

VelocityConf video

I got into programming as a career because it was easy. I could solve puzzles all day, then go home at 5:30 and drink with my friends. Twenty years later, I stay in software because it is hard. And it keeps getting harder, and more interesting.

I want to take you on that journey with me. How I moved from caring about puzzles to products, about correctness to change, about tech to the domain. And then at the very end, I’ll tell you my secret hope about how the software industry might help the world. (hint: it is not disruption)

Provisioning Puzzles

My first job was in telecom. A customer needs a phone, and to activate it they call customer service. A representative works in the customer service app, which calls out to provisioning on new activations. That was my piece. The switches need to know which phones go with which phone numbers, and we told them.

It was fun! We received requests, split them up and populated them according to instructions in our own external DSL. We forked off processes to talk to each switch — for concurrency! and failure boundaries! — and talked to them over sockets.

I had all kinds of fun puzzles. I could make new functions in the internal DSL, make new switch-talking processes. We even wrote little programs to unit test new functionality… and then threw them away.

For me, it was puzzle solving. Puzzle solving is safe and satisfying.

I knew what “correct” is because someone told me. I had defined means available: we worked in C, on Unix, with the standard library, database drivers, and Oracle. Those are the tools at hand. For bonus points, make the code look like the rest of it.

It is tempting to think this is what software development is like. We certainly interview like it is.

But I’m here now because this ability to solve puzzles in code is only a stepping stone to the real work.

Provisioning Problems

A few years later, we got a new phone company client, a new implementation of our software to customize.

This time I got to participate sooner and at a higher level, not just solving puzzles, but helping define them.

My system view included our customer, including not only the customer service reps but also the business owners who had other needs.

It needs to be Service Oriented Architecture, so we’ll introduce queuing. Now, we still had a shared database, but I didn’t know any better at the time, so I didn’t scream.

And political cover: if we’re going to replace the old system we’d better have feature parity. That’s a whole list of new requirements.

Each concern we took back to the implementation level. But now they’re not puzzles, they’re more flexible than that. It’s closer to problem-solving. I start an implementation, and if it’s not taking this system in a healthy direction, I go back to our customers, and I’m like, this feature the old system had, what do y’all use it for? “Oh, we do this one thing with it.” How about we do that one thing another way?

I zoom in and out from detail to context. Implementation informs design.

We think of the feedback loop as design, implementation, user feedback, more design.

But you, developer, are the first user of the design. And your experience informs it. Design is negotiated in this way.

And since then, since 2001 when I worked on this project, there’s even more impact, because we don’t do most of the implementation ourselves. We import libraries, we pay for software as a service.

How each of these works — how queuing works, how the database works, what a framework supports — informs our implementation and design.

But if all the fun puzzles have been solved already, what do we get to do? Build on them.


To illustrate this, I want to turn this diagram into a map. The difference is, in a map, physical location on the page has meaning. So let’s add some axes.

The vertical axis is visibility. The user is at the top, deep support at the bottom.

The horizontal axis is increasing standardization. Reusability. To the right, there is a strict definition of correct. To the left, more exploring. I can use this to talk about what is worth implementing internally. And to anchor the area of software I’m talking about.

This is called a Wardley map. It is useful for strategy.

At the far right we have utilities. Electricity is here. The network switches are over here, because they’re very well specified. From where I am, We don’t change them, and we trust them to work. The database is close to a utility, because the SQL specification is well defined, and we count on it. Best practices are well established here.

A bit more to the left, we have someone else’s product. Another company releases this library, or provides this service.

Both someone else’s products and utilities represent puzzles that have been solved for us. We can use these as building blocks for everything to the left of them. Stable dependencies. Things I love about products: what we can do with them, and what we don’t have to know.

In Why Information Grows, César Hidalgo defines a product as embodied information. A product enfolds practical knowledge about the world, in a way that we can use that knowledge without having it.

With a pencil, I have the power to write on paper, without knowing how to mine graphite or write it in wood. I get to spend my brain thinking about what letters to write.

The product teaches us how to use it, and after that we get the power. We get the power of queuing without understanding the intricacies of this distributed system.

Software as a service is the best kind of product, because it comes with a team of human experts; it takes humans to make a system resilient. Also I love that someone is making this component of my system better, without me having to think about it.

This is crucial because the limitation of our work is not how much we can do. Our work is not typing. (You can tell, because we don’t type all day, and most of what we type, we delete.) Our work is making decisions. Our limitation is how much we can know. There is a lot to know here!

Only so much fits in a human head. That’s why we have teams. And that’s why we build as much of our system as we can out of someone else’s products. Each of these, each of these nodes in this map, represents a capability. Other people’s products represent capabilities we can get with very little knowledge, and some money. Money is a lot easier to move around than knowledge.

This frees us to focus on what other companies don’t do.

Internal products

We do most of our work over here, in custom software, in systems that are specific to our company.

Over here, the capabilities also embody information. The really good software systems are like an internal product in that way.

The rest of this org has a provisioning capability to activate phone service without details of communication switches, that’s encapsulated in the provisioning software and team.

Think about what capability you’re adding to your organization. Is it something special to your business, or the context or people inside your business? Or is it a capability many businesses need? Because that common stuff, it gets abstracted out and it moves to the right over time. It belongs over here to the right, as we figure out how to do it well it goes this way.

Four years ago I worked at a biotech company. I got to work on creating an internal platform as a service on top of the utilities offered by AWS. Did the company need this? Eeehhh. Was it specific to this company? No, except for some policies encoded in the UI and default setups.

Since then, there’s been tons of PaaS pop up in Someone Else’s Products. Yet, the thing that’s really made the difference, that hit the right level of abstraction, is Kubernetes. It is not a PaaS. It is the platform that you build your platform on. So every company can have a custom PaaS, with the UI and policies that fit in that business, without building as much of the common stuff, the instances and the load balancing etc etc. Now is a much better time to build your PaaS, because big chunks of it have moved to the right.

Custom software is expensive. In money, and also in the knowledge you need to add and grow inside the company.

Why write custom software instead of buying something? we can change it. we can update the software as our business learns.

Because companies these days are made of people, policies, and software. Hell, I’m made partly of software here days. I am not the same person when I don’t have my robot brain. If the software can learn (hint: change and learn are the same thing) then the business can learn, it can get more useful to its customers.

So change is essential. More than essential, it’s the whole point. This is very different from puzzles.

See, puzzles have an end state, they have a done. I don’t want to be done! I want to keep providing capabilities to the organization. The goal state is, still playing!

See, puzzle solving is a closed game. Like a board game, or one game of baseball. There is an end state, a defined means, and a brief feeling of accomplishment.

Product development an open game. We don’t write it, we grow it. Like a career in baseball, or a game of pretend. We want to keep being useful.

And grow more useful, useful to newer to versions of the business. We have at our disposal all the puzzles that have been solved for us, and: change.

Our job is not to write software. It is to change software. And not just code! Our job is to change the system. Change gives us access to every state except this one.

Custom software gives us the potential to change more quickly than utilities or someone else’s product. But we don’t automatically get that ability. What are the obstacles to change? We know about obstacles like code quality. Lack of tests, or too many tests.

It’s easy to focus on internals because we control those. And indeed, some obstacles are there. but most of the meaningful changes are visible from outside the system — they’re new features, or better UI. And you know where the biggest obstacles to change are there? They’re outside the system.

They’re all the people and software with expectations of our software. All the systems bigger than our piece that we are really afraid to break. Software needs backwards compatibility. People need training.

Oh and then there’s the really nasty obstacles that are both internal and external: like when customer service reports reach into our database tables. Suddenly even our internal parts are frozen in fear of breaking systems more important than ours.

This is a process! We are not designing static code. We are designing change.

Designing change is: where is the next increment of where we want to go?

How will the whole system get there? This means data migrations, deprecation, feature flags. Documentation, versioning. How will we bring the adjacent systems along? Get them to use the new features? Get them to upgrade auth so we can improve security. We need to do advocacy with our users.

How will we know whether it had the effect we wanted, and what other effects it had?

With puzzles, if nobody uses it, whatever, I did my bit. With products, with designing change, we care very much.

A positive is: When you think of code this way, feature-flag if-statements aren’t ugly. It’s OK if your code isn’t at the tip of the target architecture. It is the movement that matters. Deprecation is beautiful. Show me the movement.


Movement is harder the more users you have, and the more distant your interactions with them. Look at utilities, who are used by everyone, and everyone has expectations. When S3 goes down, the world has problems.

The bigger the system, the slower the timescale it moves at. Utilities move at the timescale of the industry. In custom software, we can move at the timescale of the one company, if it’s centralized, or the team, if it’s decentralized.

So think about the timescale of change you need. Industry? Company? Team?

For instance, take delivery. Since “change is our job”, delivery is crucial. Designing change only starts with a code push.

Delivery is custom, because it’s contextual. We can’t wait on any other company to accommodate both our existing architecture targets and our next one.

Fortunately, parts of delivery are common. Everybody needs triggering, needs the new code cloned. You don’t need to figure out how to do that: use someone else’s product. Save your knowledge for what is common across the company (run security checks) and specific to a team (format the code). Whatever’s centralized can move at the speed of the company, which can be faster than an external product but will be slower than the speed of a single team. So decentralize what you can.

Think about where the industry is on any particular puzzle solution. Where you can go to other people’s products, where you’re doing something specific to your business, where you need control.

Because control is expensive. Custom software is expensive, and it’s extremely hard.

Hard like rocks, hard like mud

Custom software, growing products, is incredibly hard. Yet “hard” is not specific enough. Solving puzzles is hard. It’s intellectually challenging, it’s work, and it feel satisfyingly productive at the end of the day. It’s hard like rocks. Rock climbing is very technical, and very hard work. And when you get to the top, you feel great! And then what? You go back down. You go home.

Growing a product is hard in a different way. It’s dealing with people, with ambiguity, with probabilities instead of absolutes. It’s hard like mud. Like wading through goopy mud.

Why would you want to wade through mud?

Why does a fish want to swim through water? It’s all around us. Politics, interrelationships, context, ambiguity — this is the world we swim through. If we acknowledge this and work with it, we can add real value, make real change.

We can also find surprises.

At the extreme of wading through mud, beyond finding answers to questions that don’t hold still, there’s finding whole new questions. That “genesis” phase over here on the left of the Wardley map. Here is more ambiguity. Here we can deliver something faster than we can define it. Every time you find an idea you need to bring it back to reality with some sort of test. And then let go of it.

I get to work here now, with our CEO, Rod Johnson. Using the capabilities in Atomist’s Custom column, we’re exploring, what kind of questions can we ask about the condition of code across hundreds of repositories at once?

Once we find a promising problem to solve, then it moves into custom development, where we make it real at scale and connect it with everything else. That is serious work, growing this idea into a product.

This is where I want to be most of the time.

Why move into growing products? Why choose to wade through mud?

The mud is the reality of your business, and this is where you are most valuable to the company. This is where we need the real talented developers, in the core domain that is this company’s secret sauce. The highly specific, challenging to nail down, supple domain work.

Talented developers like to move to the more general, abstract infrastructure projects where they can find some hope of defining “correct.” This gives them status among developers. Gold stars on GitHub. Stuff they can speak about at conferences because it’s not secret, because any company might have these same problems.

In the business domain, you can find the shortest path to meaningful work. I define “meaning” as activity that has significance at higher levels of the system you identify with. Identify with the business you’re part of. This gives you better decision-making power than if you identify generically as “a software craftsman.” You can zoom out to context of use, rather than resorting to generic “code quality” mantras.

This can be very satisfying, if the situation is right.

You need:

  • Visibility into impact. Who is using your product, and which parts? How is it working for them? Are they using it more over time?
  • Efficacy. The work you do goes somewhere. This is why I like working on software that’s already in production.
  • Autonomy. Your team chooses the changes to design and how to implement them, with input from others. Be easy to convince, but be convinced. If you take someone else’s word for what needs to be done next, then you aren’t doing it for the right reasons. You won’t be able to make the decisions correctly, because you don’t have the “why.” You can’t zoom out past “so and so told me to.”

How do we become good at growing products?

  • Be ready to be wrong. Eager, even. “Correct” is the enemy of “better.”
  • Get good at zooming in and zooming out
  • Seek out stories about the history of the software and the business.
  • Make friends who know the business. make friends all over the company.
  • Think about deltas. Design change, not code.
  • Be accountable to tell a story. Not for outcomes – there are a million influences on that KPI besides what you’re doing, and what you’re doing influences a million things besides the API. (Please care about those things.) Tell the story of your team’s decisions and what happened, so the whole organization can learn.Your intentions are not revealed by any one thing you do, but by your responses to the world’s responses to what you do. Action is a process.

I don’t know all the technology, but I know something about the business I’m in.I can’t find the solution, but I’m learning to design change.I can’t solve the puzzle, but I can wrestle with problems.

In software, we get the chance to change complex systems at a rate humans never could before. We get the opportunity to learn about complexity, in a laboratory where we can inject clues for ourselves, to find out how it’s working.

My secret hope is: if we can get good at designing change in a system too complex to understand, then maybe that will help us, as a species, to change our world without destroying it.

References & Further Reading

  • Products and knowledge as a limitation: Why Information Grows by César Hidalgo
  • Closed and open games: The Grasshopper, by Bernard Suits
  • Infinite games: Finite and Infinite Games, by James Carse
  • If the business doesn’t believe in growing products: Project to Product, by Mik Kersten
  • John von Neumann and Norbert Wiener, by Steve J Heims
  • How Reason Almost Lost Its Mind, by Erickson, Klein, Daston, Lemov, Sturm, Gordin

A definition of play, and how to live

In games, we choose an objective that has no intrinsic value. Get points, run out of cards, reach the finish line. We take aim, and restricting our actions with rules, because this leads us to actions that we enjoy. Thinking, interacting with other players, running all-out. We play the game because it’s fun. We try to win because that makes it fun. (Some people get happy if they win. But if you don’t enjoy the play, you won’t keep coming back.)

This is play, because the ends don’t have particular value, but the means of getting there give us satisfaction.

We can take this strategy in life too:

Choose the ends that lead you to the means that get you what you need.

I call it a quest, an unreachable star, this aim that we choose not because we expect to get there (that would be a milestone) but because it leads us in useful directions.

The book Obliquity (amazon) explains it well: there are some things we can’t get by aiming for them, such as profit or happiness. So you choose an end (“build the best airplane”) that leads you to means (engineering, research, investment, production quality) that get you what you need in order to keep going (profit). Choose an end (“build up my community”) that leads you to means (forming relationships, organizing, helping people) that get you what you need (joy).

When the end has some intrinsic value of its own, like the airplane, or the community, or operating useful software — then we call it work.

People say “Do what you love.” This is how to do that: find an objective that matters to others, which also leads you to means that bring satisfaction. Some people find hard physical work satisfying, others mental exertion, others human interaction. It doesn’t need to be your favorite activity; fun does not equal joy.

When both the ends and the means are fulfilling, then work and play align.

Each milestone (produce an engine, get someone to like you, code up a feature) has many routes to reach it. If you aim for the quickest route, you might end up messing up your quest (the fastest code is harder to operate) or worse, missing out on what you need (long-term profit is down, the community is poisoned, the work is unsatisfying). How do we restrict our means to the ones that take us toward our quest, not just our milestone? and also give us what we need to keep going?

In games, we use rules. In life, we use values.

Change at different timescales

On recommendation from @mtnygard and others, I have acquired a copy of How Buildings Learn (Stewart Brand, 1994). Highlights so far:

Buildings are designed not to adapt, but they adapt anyway, because the usages are changing constantly. “The idea is crystalline, the fact fluid.” They’re designed not to adapt because “‘Form ever follows function’ … misled a century of architects into believing that they could really anticipate function.”

We think of buildings as static, because they change at a timescale slower than our notice. They change over generations. Humans operate at some natural timescale, and we see things slower as unchanging, and things faster as transient, insignificant. We are uncomfortable with change in stuff we think of as permanent, like buildings or culture or language. It isn’t permanent, just slower than us.

A jar of honey on its side will leak. The lid doesn’t trap the honey, just slows it down. Imperceptible movement is invisible, until the whole cupboard is sticky.

Software’s pace of change can be unnaturally fast, the opposite of buildings. That makes people uncomfortable. Updating buildings, “we deal with decisions taken long ago for remote reasons.” In software, “long ago” might be last year.

As usages change, so must our environs, in brick and in computers.

What changes faster than usages? Fashion. “Buildings are treated by fashion as big, difficult clothing, always lagging embarrassingly behind the mode of the day. This issue has nothing to do with function.” The latest hot tech may not improve on the value of your legacy software. “The meaningless change of fashion often obstructs necessary change.”