Developers are system changers

Some people work in a system, and some people work on a system.

Like, you can be the person who washes the dishes, or the person who installs and maintains the dishwasher.

You can be the person who assembles the reports every week, or the person who automates that report assembly. (Jacob Stoebel told this story on >Code #148 today. That’s how he got into software development.)

You can conform to a system, or you can participate fully — part of serving the system is changing the system to better serve you.

Developers are inherently system changers. That’s what we do. No wonder we’re hard to manage!

No wonder software communities are full of turmoil and rabble-rousing and shifting technologies: we are a whole industry full of system changers.

Also on >Code today, we talked about personal automation. Chante Thurmond remarked on the tools that exist today to let people (not just developers!) customize, tweak, and automate their work. We can all craft the systems we operate in. More and more system changers.

This is the real change software makes in the world.

To scale it up, smooth it out

To scale up the system you’re changing, scale down the changes.

The bigger and more complex the system you’re working on, the more people involved, the more affect a change can have — the smaller each change needs to be.

When change is expensive, when it’s scary, don’t make fewer changes. Work to make changes less expensive (no matter how expensive that work is), so that everyone can make smaller changes, because tiny changes aren’t so scary.

Commit code sooner, work in small batches, get the code out there before the merge conflicts erupt. The State of DevOps Report for 2019 remarks that this helps in open source software. And that in business, a heavyweight change process makes everything worse, including the very risk it was supposed to reduce.

Economies of scale are reversed in software. One big change is more expensive than many small changes. To get bigger, get smaller.

(see: Kent Beck on Limbo and scaling software beyond current imaginings)

When the db is the interface

There are two huge sources of inertia in software: data, and interfaces.

When two systems connect to the same database, you combine both. Ow.

When some other system is doing reporting on my database, I can’t change my table structure. That severely limits any internal reorganizations I might do. Especially when I don’t know how they use it, so I am afraid to touch it.

Then what if it’s a schemaless database? This means the schema resides in the application. When the schema resides in two applications at once, designing change is an exercise in hope.

Sometimes two systems access the same database because the database is the interface. We have an example of that at Atomist currently: one system (called squirrel, I don’t know why) populates a database, and another system (I call it org-viz) uses that data to make visualizations. How can that be okay?

Database as interface is not horrifying when:

  1. The database is not the system of record. We could repopulate all or part of it from elsewhere, so it’s somewhat disposable.
  2. The database has a schema. Interfaces must be well-defined.
  3. One system updates, the other reads.

In our case, we use PostgreSQL, so we have a schema. Some of our fields are JSON, which gives us extensibility, while the essential fields are guaranteed. The populating system (squirrel) operates from a queue, and we can replay the queue as needed, or play it into multiple databases. There are options for designing change.

Database as an interface is never going to be the cleanest decoupling, but it is not unreasonable when it is carefully designed and the teams don’t mind talking to each other. When the database is accidentally an interface, then you’re horked.

Inertia in the interface

What makes software hard to change?

As a developer, it’s easy to focus on the internal properties of the software system. The code needs refactored, the framework is old, we need more tests, or else fewer tests.

If your software is in production, these are not the biggest obstacle to change.

The important changes are the ones visible to the outside world, the ones that change the interface. And changing the interface means more than your software has to change: the systems that use your software need to change.

The more useful your software is, the more other systems depend on it, the scarier change is. You’re risking more than your piece of the world. You need progressive delivery, careful data migrations. Backwards compatibility: twice as many tests, and special cases in the code. You need to design the whole change.

And then, for your work to be useful, people have to use it. You must get the word out through social channels: talking to people, advocacy, (dare I say it) marketing.

This is all fine. It is right. No complaints.

Because our work is not to change software, it is to change a system. To add and broaden capabilities in systems larger than ours.

If our code is full of if/else statements for backwards compatibility, if our database integration is cluttered by a migration, if we spend more time crafting the deploy strategy than we did writing the code, cool.

The more people use the code, the harder it is to change, and the more valuable.

Caveat: yes, there are performance and scale improvements that are useful. A wider variety of functionality changes are useful, and these are more frequent, so that’s what I’m talking about here.

For more on this topic, see: From Puzzles to Products

Change at different timescales

On recommendation from @mtnygard and others, I have acquired a copy of How Buildings Learn (Stewart Brand, 1994). Highlights so far:

Buildings are designed not to adapt, but they adapt anyway, because the usages are changing constantly. “The idea is crystalline, the fact fluid.” They’re designed not to adapt because “‘Form ever follows function’ … misled a century of architects into believing that they could really anticipate function.”

We think of buildings as static, because they change at a timescale slower than our notice. They change over generations. Humans operate at some natural timescale, and we see things slower as unchanging, and things faster as transient, insignificant. We are uncomfortable with change in stuff we think of as permanent, like buildings or culture or language. It isn’t permanent, just slower than us.

A jar of honey on its side will leak. The lid doesn’t trap the honey, just slows it down. Imperceptible movement is invisible, until the whole cupboard is sticky.

Software’s pace of change can be unnaturally fast, the opposite of buildings. That makes people uncomfortable. Updating buildings, “we deal with decisions taken long ago for remote reasons.” In software, “long ago” might be last year.

As usages change, so must our environs, in brick and in computers.

What changes faster than usages? Fashion. “Buildings are treated by fashion as big, difficult clothing, always lagging embarrassingly behind the mode of the day. This issue has nothing to do with function.” The latest hot tech may not improve on the value of your legacy software. “The meaningless change of fashion often obstructs necessary change.”

Inner coherence vs usefulness

Around 1900, when modernist art was emerging, art historians talked about the significance of art in context: the painting is not complete without the beholder.

Before that, the beholder wasn’t so important. People looked for art to express some universal truth through beauty.

Before cultures and artists considered the role of the beholder, they made art that didn’t need you. The art has inner coherence.

A lot of software development aims for inner coherence. Code that is elegant, that is well-designed and admirable on its own.

I used to like that, too. But now I want to think about code only in context. Software is incomplete without use.

If my code is full of feature flags and deprecated fields for backwards compatibility, that’s a sign that it is used. The history of the software is right there to see. I don’t want to hide that history; I want to make it very clear so that I can work within it.

My job isn’t to change code, it’s to change systems, so that we can adapt and grow increasingly useful instead of obsolete.

This old painting may be beautiful, but it doesn’t affect me the way Gustav Klimt’s work does. (some drawings, NSFW) He was one of the early modernist artists to speak of contextual, rather than universal, truths.

Within our teams, contextual truths have the most power. In my software, it’s a contextual coherence of its larger system that I care about.

Tiny dramas, tiny deploys

It is better to practice risky things often and in small chunks with a limited blast radius, rather than to avoid risky things.

Charity Majors, “Test in production? Yes

Charity is writing about deploys. Not-deploying may be safer for tonight, but in the medium term it leads to larger deploys and bigger, trickier failures.

In the long term, slow change means losing relevance and going out of business.

In relationships, the same applies. If I have some feeling or fact that my partner might not like, I can say it or not. It never feels like the right time to say it. There is no “right time,” there is only now. There is positive reinforcement for holding back, because then our evening continues pleasantly. No drama.

This leads to an accumulation of feelings and facts they don’t know about. Then when it does become urgent to talk about those, they react with feelings of betrayal: Why didn’t you tell me about this sooner?

In the long term, lack of sharing means growing apart and breaking up.

My new strategy in relationships is: tiny dramas, all the time. The more tiny dramas we have, the fewer big dramas. Also we get practice at handling drama in a way that is safe, because it’s minor. I take any mental question of “should I say this?” as a clue, an opportunity! Yes, say it. Unless it’s a really bad time, it’s the best time.

And the complementary strategy: whenever my partner tells me something scary, like something I did that they don’t like or some feeling they had that might upset me, my first response is “Thank you.” Usually it is not a drama anyway, it’s fine. When I do have feelings about it, we can talk about them. Reassurance helps a lot, especially when I recognize and appreciate the risk they took by telling me in this moment.

If a small deploy causes failure, please respond with “Thank you for not making this part of a bigger deploy.”

We have built a glass castle, where we ought to have a playground.

Charity again, on our lack of safe tooling and therefore fear of production

Designing Change vs Change Management

Our job as developers is to change software. And that means that when we decide what to do, we’re not designing new code, we’re designing change.

Our software (if it is useful) does not work in isolation. It does not poof transition to a new state and take the rest of the world with it.

If our software is used by people, they need training (often in the form of careful UI design). They need support. hint: your support team is crucial, because they talk to people. They can help with change.

If our software is a service called by other software, then that software needs to change to, if it’s going to use anything new that we implemented. hint: that software is changed by people. You need to talk to the people.

If our software is a library imported by other software, then changing it does nothing at all by itself. People need to upgrade it.

The biggest barriers to change are outside your system.

Designing change means thinking about observability (how will I know it worked? how will I know it didn’t hurt anything else?). It means progressive delivery. It often means backwards compatibility, gradual data migrations, and feature flags.

Our job is not to change code, it is to change systems. Systems that we are part of, and that are code is part of (symmathesy).

If we look at our work this way, then “Change Management” sounds ridiculous. Wait, there’s a committee to tell me when I’m allowed to do my job? Like, they might as well call it “Work Management.”

It is my team’s job to understand enough of the system context to guess at the implications of a change and check for unintended consequences. We don’t all have that, yet. We can add visibility to the software and infrastructure, so that we can react to unintended consequences, and lead the other parts of the system forward toward the change we want.