Definition of DevOps

Step 1: On one team, put the people with the knowledge and control necessary to change the software, see the results, change it, see the results.

Step 2: Use automation to take extrinsic cognitive load off this team, so that it needs fewer people.

That’s it, that’s DevOps.

Step 1 describes the cultural change that leads to flow. Delivering change requires no handoffs or approvals from outside the team; the impact of change flows back to the team. Act and learn.

Step 2 is where tools come in. If all you do is improve your tooling, well, that helps a little, but it doesn’t get you the qualitative change in flow. That comes from Step 1. The serious value of automation is that it enables Step 1, a single team with all the relevant knowledge.

Our job as developers is making decisions. DevOps gets us the knowledge we need to make good decisions, the authority to implement them, and the feedback to make better ones in the future.

Development aesthetic: experiments

Software development is a neverending string of “What happens if I…?” Each new runtime, language, or tool is a new world with its own laws of nature. We get to explore each one with experiments.


Today I added another alias to my PowerShell $profile:

echo "Good morning!"
# ...

Function ListFilesWithMostRecentAtBottom {
    Get-ChildItem | Sort-Object -Property LastWriteTime
Set-Alias ll ListFilesWithMostRecentAtBottom

To use that alias in my current shell, I need to source the profile again. I googled how to do this. The first useful page said:

& $profile

So I typed that. It echoed “Good morning!” but the alias did not work.

Hmm, did it not save?

I can test that. I changed the echo to “Good morning yo!” and tried again.

It printed the new text, but still didn’t get the alias.

Hmm, is something wrong with the alias?

I opened a new shell window to test it.

The new alias works in the new window. Therefore, it’s the & $profile command that is not doing what I want.


I could ignore the problem and continue work in the new window. My alias is working there. But dang it, I want to understand this. I want to know how to reload my $profile in the future.

Time for more googling. The next post had a new suggestion:

. $profile

I typed that, and it worked. yay!

But wait, was that the old window or the new window? What if it only worked becuase I was in the new window?

I want to be certain that running . $profile brings in any new aliases I just added. For a proper experiment, I need to see the difference.


I add a new alias to my $profile, and also change the echo so that I’ll be sure it’s running the new version.

echo "Good morning yo tyler!"
# ...

Function ListFilesWithMostRecentAtBottom {
    Get-ChildItem | Sort-Object -Property LastWriteTime
Set-Alias ll ListFilesWithMostRecentAtBottom
Set-Alias tyler ListFilesWithMostRecentAtBottom

In my terminal, I run tyler as a test case, then the command I’m investigating (. $profile), then the test case tyler again.

Now I can see the before and after, and they’re different. I can tell that . $profile has the desired effect. Now I have learned something about PowerShell.


I remove the extra tyler stuff from $profile.

As far as I can tell, & runs the script in a subshell, and . runs the contents of the script in the current shell. The . command works like this in bash too, so it’s easy for me to remember.

Today I took a few extra minutes and several extra steps to make an experiment and figure out what PowerShell was doing. Now I know how to reload my $profile. Now you know how to run a tiny experiment to ascertain that what just happened, happened for the reason you think it did.

Don’t just keep trying; report your limits

The other day we had problems with a service dying. It ran out of memory, crashing and failing to respond to all open requests. That service was running analyses of repositories, digging through their files to report on the condition of the code.

It ran out of memory trying to analyze a particular large repository with hundreds of projects within it. This is a monorepo, and Atomist is built to help with microservices — code scattered across many repositories.

This particular day, Rod Johnson and I paired on this problem, and we found a solution that neither of us would have found alone. His instinct was to work on the program, tweaking the way it caches data, until it could handle this particular repository. My reaction was to widen the problem: we’re never going to handle every repository, so how do we fail more gracefully?

The infrastructure of the software delivery machine (the program that runs the analyses) can limit the number of concurrent analyses, but it can’t know how big a particular one will be.

However, the particular analysis can get an idea of how big it was going to be. In this case, one Aspect finds interior projects within the repository under scrutiny. My idea was: make that one special, run it first, and if there are too many projects, decline to do the analysis.

Rod, as a master of abstraction, saw a cleaner way to do it. He added a veto functionality, so that any Aspect can declare itself smart enough to know whether analysis should continue. We could add one that looks at the total number of files, or the size of the files.

We added a step to the analysis that runs these Vetoing Aspects first. We made them return not only “please stop,” but a reason for that stop. Then we put that into the returned analysis.

The result is: for too-large repositories, we can give back a shorter analysis that communicates: “There are too many projects inside this repository, and here is the list of them.” That’s the only information you get, but at least you know why that’s all you got.

And nothing else dies. The service doesn’t crash.

When a program identifies a case it can’t handle and stops, then it doesn’t take out a bunch of innocent-bystander requests. It gives a useful message to humans, who can then make the work easier, or optimize the program until it can handle this case, or add a way to override the precaution. This is a collaborative automation.

When you can’t solve a problem completely, step back and ask instead: can I know when to stop? “FYI, I can’t do this because…” is more useful than OutOfMemoryError.

Align the stars (programmatically)

Yesterday I was ready to get some changes into master, so I merged in the latest and opened a PR. But NO, the build on my pull request broke.
The error was:

ERROR: (jsdoc-format) /long/path/to/file.ts[52, 1]: asterisks in jsdoc must be aligned
ERROR: (jsdoc-format) /long/path/to/file.ts[53, 1]: asterisks in jsdoc must be aligned
ERROR: (jsdoc-format) /long/path/to/file.ts[54, 1]: asterisks in jsdoc must be aligned
ERROR: (jsdoc-format) /long/path/to/file.ts[55, 1]: asterisks in jsdoc must be aligned
… fifty more like that …

gah! someone added more tslint rules, and this one doesn’t have an automatic fix. Maybe someone upgraded tslint, its “recommended” ruleset changed, and bam, my build is broken.

For measley formatting trivia.

* oh no! This JSDoc comment is not aligned perfectly!
* The stars are supposed to have one more space before them
* so they all line up under the first top one
* The world will end! Break the build!
* @param likeItMatters a computer could fix this grrr
* @deprecated
whatever(likeItMatters: Things): Stuff;

Look, I’m all for consistent code formatting. But I refuse to spend my time adding a space before fifty different asterisks. Yes I know I can do this with fewer keystrokes using editor acrobatics. I refuse to even open the file.

You know who’s good at consistency? Computers, that’s who. You want consistent formatting? Make a robot do it.

tslint robot: “Your Code Is Not Acceptable!” / me: “I have better things to do :-(” / Atomist: “I will save you!”

So I made a robot do it. In our Software Delivery Machine, we have various autofixes: transformations that run on code after every commit. If an autofix makes a change, the robot makes a commit like a polite person would. No build errors, just fix it thanks.

I wrote this function, which does a transformation on the code. The framework takes care of triggering, cloning the repository, and committing the result.

const alignAsterisksInProject: CodeTransform = (project: Project) =>
doWithFiles(project, "**/*.ts", async f => {
const content = await f.getContent();
if (hasUnalignedAsterisks(content)) {
await f.setContent(alignStars(content));

You can do this too. You can do it on your local machine with the fully open-source Local Software Delivery Machine. Fix your code on demand, or in response to each commit. Write your own functions to keep your code looking the way you like. Never be embarrassed by the same mistake again!

To help you try it out, I added my autofix to an otherwise-empty Software Delivery Machine.

Align your own stars

Try it yourself (on Mac or Linux, currently):

  • Install the atomist command line: npm install -g @atomist/cli@next
  • By default, atomist works on projects in ~/atomist/projects. If you prefer a different directory, create one and set ATOMIST_ROOT=/path/to/that/directory
  • Bring down the code for the Software Delivery Machine (SDM): atomist clone 
    (This does git clone, plus sets up git hooks for magical autofixes.)
  • Go to that location: cd $ATOMIST_ROOT/atomist-blogs/align-stars-sdm
  • Now for the slow part: npm install
  • Start up your SDM: atomist start --local
    The SDM is a process that hangs out on your computer waiting to help. It swings into action when triggered by the atomist command line, or by commits inside $ATOMIST_ROOT.
  • Optional: in another terminal, run atomist feed. This will give you a running summary of what your SDM is up to.
  • Now screw up some formatting. I’ve left some nice jsdoc comments in lib/autofix/alignStars.ts; move those stars over a little.
  • Save and make a commit: git commit -am “Oh no, misalignment”
  • Check the output in your atomist feed, and you should see that an Autofix has run. (You can also type atomist align the stars to do this specific transform, in a repository not wired up to trigger you SDM.)
  • Check your commit history: git log. Did Atomist make a commit? Check its contents: git show and you should see the stars moved into alignment.
  • (if not, please leave me an issue or ping me (jessitron) in community slack)

But wait! That’s not all.

OK that was a lot just to format some comments. The important part is we can write functions to realize policy. If a person complained about this formatting I’d tell them to f*** off or fix it themselves — in fact I did curse extensively at tslint when it gave me these errors. Tslint didn’t care. Programs aren’t offended.

If I want my teammates’ code to conforms to my standards, it’s rude to ask them to be vigilant about details. Aligning asterisks — I mean, some people like it, and that’s fine, I kinda enjoy folding laundry — but as a developer that’s a crummy use of my attention. Computers, though! Computers are great at being vigilant. They love it. My Atomist SDM sits there eagerly awaiting commits just to dig around in those asterisks in the hope of fixing some.

Please make my star-alignment into a code transform that’s useful to you. I went with plain string parsing here, in a very functional style for my own entertainment. We also have clever libraries for working with the compiled AST and more (watch this space).

There’s more: an SDM running in the cloud listens to GitHub (GitLab, BitBucket, GHE) and applies autofixes to everyone’s commits. And code reviews. And runs or triggers build (but only when it’s worthwhile; it looks at what’s changed). And initiates deploys (except it asks us first in Slack). There’s no setting up a pipeline for a new repository or branch; our SDM knows what to do with a code push based on the code in it.

There’s more: an SDM in the cloud also listens to issues, pull requests, builds, deploys, and other events in the software domain. It can react to all of them by talking to people in Slack, or running any other program. Whatever we do that is boring, we can automate.

This is our power as software developers. We don’t need someone to write a GUI we can click in. We don’t need to configure in YAML. We can specify what needs to happen declaratively in code.

All we needed was a framework to do the common glue-y bits for us, like triggering on events, cloning repositories, passing our functions the data they need and carrying out our actions.

The autofix in this example triggers on commit, clones the repository, passes a Project object in for a code transform function to act on, commits those changes and pushes them back to where I’m working. The framework of the SDM lets me define my own policies and implement them in code. Then the machinery of the SDM keeps them running, locally (open source) or team- or organization-wide (using Atomist’s API).

Aligning the stars is only the beginning.

Get happy developers to do what you want

(originally published in the SD Times)

How do we influence the operation of a complex dynamical system like a team? How do we get a development team to function closer to our preferences, and still let people operate as self-actualized humans?

In Dynamics in Action, Alicia Juarrero describes complex systems such as people in terms of the constraints they operate within. Constraints make things interesting, and constraints allow for emergence.

Constraints here aren’t hard rules. They aren’t restraints, holding people still. Rather, constraints are changes in the probability of particular actions.
They’re like … an incentive landscape.

For instance: if you force people to change their password frequently, they’ll use a terrible password. If coffee is free and juice costs money, people will drink more coffee. If the stairs are pretty and easy to access, people will take them more often than the elevator.

These are all ambient (context-free) constraints. They don’t restrain the team from taking the elevator, but they do change the probability of whether each person, on each occasion, will do so.

restrain: one allowable path. constrain: one prettiest path

As a software organization — maybe as an architect, or technical lead, or SRE, or infrastructure team — how do you get developers to code in a certain way? to be consistent with the rest of the organization, to follow your 
preferred practices? Getting developers to do anything is supposed to be like herding cats.

Change the cost, change the behavior.

Development is a creative undertaking. Restraints don’t sit well. But if you make something easier for me, that’s a constraint I can live with. It will change my probabilities.

I do this for myself: a script to make submitting a single change as a PR into one step, and now I less frequently lump that function rename in with other unrelated changes.

My team expects me to look at every change coming in to the repositories we work on together, in many time zones. That seemed onerous, until I made my “git fetch and move my default branch pointer” script also open a GitHub compare window. Now this feels normal, and I wouldn’t work without this routine.

When we work in computers, a new automation changes our constraints, and this changes the probabilities of our various behaviors.

I can affect other people in my organization: I want people to improve the descriptions in our GraphQL schema, so I made an automation that deploys the schema when they push a button in Slack. I made the button appear whenever someone changes the schema in GitHub. They use this all the time.

When my coworker wants the dependencies alphabetized: sorry, I don’t care enough. He can add an autofix — a TypeScript function in our software delivery machine — to do that.

I can change the incentive landscape by making the way I want people to make software, the easy way to make software. Sometimes by giving them a button, other times having a robot come behind and clean up every push with an autofix. The autofix has the added benefit of keeping the change I want in the system, which is often harder than making the change.

Contrast this with code metrics as an incentive: metrics distract from our collective goal of making this particular software great. They draw our attention to numbers, instead of customers. Metrics are great for information, but as soon as they’re used to alter the incentive landscape, they distort it in unintended ways.

Whether for yourself or the other developers in your organization, consider automation a tool to constrain (influence) the way developers work, without curtailing freedom. Increase developer happiness and consistency at the same time!

Collective problem solving in music, art, science, and software

Or: the Origins of Opera and the Future of Programming.

(video, or TL;DR)

At the end of this post is an audacious idea about the present and future of software development. In the middle are points about mental models: how important and how difficult they are. But first, a story of the origins of Opera.

Part 1: Examples

The Camerata

The Florentine Camerata were a group of people who met in Florence in the 16th century. They had a huge impact on history and on each other.

(Caveat: I’m not a student of music or music history. I listened to this Great Course and heard about this Camerata and noticed a resemblance to some teams I’ve seen and been in. So I looked it up further, and came upon Collective “Problem-Solving” in the History of Music: the Case of the Camerata by Dr. Ruth Katz in the Journal of the History of Ideas, and toward the end of it my mind was blown. More on that later.)

Camerata literally means a small orchestra or choir. This Camerata was a diverse group of people who gathered and worked on a common problem: they were bored with polyphony, the esteemed music of their day. (Sample: Palestrina) Polyphony is very pretty: it has around four melodies, each of equal importance. Each has a logic of its own, rather than melody and accompaniment. Polyphony is intellectually rewarding, but you need technical understanding to appreciate it fully. What feeling it conveys comes through auditory qualities.

The Camerata asked the revolutionary question: what if you could understand the words?


The Camerata included (all quotes in this style are from Katz)

musicians, artists, poets, astrologers, philosophers, scientists
who met informally under the aegis of Bardi and Corsi.

People with diverse skills and perspectives worked together. They had sponsorship; Giovanni d’Bardi was Count of Venoa, and loved to surround himself with interesting people.

Their aim was to reform the polyphonic music of the day
and they believed that the best way to do so was to renovate the ancient Greek practice of setting words to music

They shared a common goal; they were unsatisfied with what Vincenzo Galilei (member of the Camerata, father of Galileo) called “that corrupt and incomprehensible contemporary music.”

And they had a common strategy. They didn’t really know what the Greeks did, but this lent legitimacy to their ideas. Like citing computer science papers from the 70s.

Their real high-level objective was horizonal, and more specific than moving away from polyphony:

Their principal aim was to find the optimum formula for wedding words and music.

Here, “optimum” is measured as “maximally effective in communicating… the specific meanings and emotions appropriate to the text.”


The Camerata talked a lot, and listened to each other talk.

“I learned more from their learned discussions than I learned from descant in over thirty years” — Caccini, renowned singer

(I had to look up “descant.” It means long-winded discourse. Like you’re experiencing now, sorry.)

But they weren’t all talk.

the Camerata constituted not only a forum for theoretical discussions, but also a workshop, a “laboratory” for the creation and performance of music.

They practiced together! And experimented. The beginning of the scientific method is a big part of the Renaissance, and it intertwines with art. We have a new way of thinking, ways of asking the universe questions. Vincenzo Galilei varied lengths and tensions of strings and found the ratios that make chords.

The Camerata didn’t always get along. There was rivalry between Bardi and Corsi, the two chief sponsors. Bardi preferred talking, Corsi wanted to play more music. These feed each other. There was disagreement over style between Caccini and Peri, the two musical stars. Peri wanted to focus on the words, with a bit of music; Caccini wanted the singing to stand out, while also understanding the words. These tensions lead to a balance.

They did code review!

presentations made… were commented on formally by “defenders” and “censors” who were nominated for the occasion.

I like that criticism came from people designated to the role, not the asshole who doesn’t like you and takes it out on your code. (Technically, this took place in the Alterati, another meetup with a lot of the same people.)


Over the years, this team changed history. They invented the music-drama, and a style of music that conveyed more meaning. (Sample: Monteverdi, a first composer to adopt the Camerata’s style. If you know Italian, you can probably understand the words.)

What about the individuals? Their outcomes were exceptional too! Here are some of their publications:

As composers of operas and authors of scientific treatises, these half-dozen people are fewer than half of the Camerata members who have Wikipedia articles. Really, what are the chances, if you’re alive in the sixteenth century, that you have a Wikipedia article today? These people did pretty well for themselves. They grew out of the Camerata.

Also in Science

This pattern of a group of people coming together to solve a problem is not unique to music — it’s the common case.

the Camerata resembles the kind of “invisible college” which is the key to creativity in science.

This “invisible college” is an association of people who share ideas. Who build a new reality together, then spread it to advance the wider culture.

We like to give the Nobel Prize to one or two people. But who worked in their lab? Who did they correspond with?

When Jon Von Neumann went to Los Alamos for the Manhattan Project, so did two or three mathematicians that he went to high school with. Really, you grow up in Hungary, what are your chances of getting to Los Alamos? They built each other up.

These invisible colleges share:

tacit understandings concerning appropriate methods of research

(processes and values)

priority problems

(this means they fight over who was first; more on this later)

and the shorthand communication which shared work implies.

We can move quickly together because we share common ground, compatible mental models. This is super fun, when I get to this point with my team.

Also in Art

People work together to develop their individual styles. Usually in Paris, it seems.

the salon, the coffeehouse, the café as breeding places of artistic creativity

In the nineteenth century, a group of artists broke from the mainstream and developed Impressionism.

coping with a common puzzle which they, separately and as a group, tried to solve

Van Gogh lived in the Montmartre district with the other artists and dealers and critics. When I visited his museum in Amsterdam, my favorite part was all the paintings by his friends and associates; they developed each other as painters.

One of these (my personal favorite) was Paul Gauguin, the one Van Gogh cut his ear off over. Gauguin went on to influence Picasso.

Picasso was at the center of many social circles in Paris over the decades. Writers, photographers, philosophers.

One painter who dipped in and out of his camerata was Aleksandra Ekster, who took the ideas of Cubism back to Kiev, where her studio was its own place of idea exchange.

One of her high school classmates and friends was Alexander Bogomazov, and a print of his lives on my bedroom wall.

my own little museum. includes “Head” by Bogomazov next to a piece by my daughter

This brings us to the modern day, where we can find examples of this phenomenon in software teams.

Also in Software

One camerata emerged in London around 2003–2006. This group gave us Continuous Integration, Continuous Delivery, DevOps. Many of these people worked at ThoughtWorks. They weren’t all on the same project, but they talked to each other, and they solved the problem of: Does deployment have to be so hard?

Jez and Dan and Chris Read produced The Deployment Production Line. Later, Dan went to invent BDD, Sam Newman became a prophet of microservices, and more. I keep meeting conference speakers who were part of this group.

Another example: the early Spring team, around the same time. They came together online, from all around the world, to solve the problem of: do we really have to use J2EE? and made Java development less painful for everyone. Today, Java development is (approximately) everywhere, and Spring is everywhere Java is.

That group of developers and businesspeople produced an inordinate number of founders and CEOs and partners.

Rod Johnson, CEO Atomist; Christian Dupuis VP Eng, Atomist; Steven Schuurman, Founding CEO, Elastic; Neelan Choksi, President and COO, Tasktop; Rob Harrop, CEO, Skipjaq; Adrian Colyer, Venture Partner, Accel

Personally, I’ve been part of three teams that grew me as a developer and as a person. The tech meetup scene in St Louis is a source of growth too. We have several groups of about twenty, and a lot of overlap so we can talk to the same people multiple times a month about interesting things.

In all of these examples, we can observe a phenomenon:

Great Teams Make Great People.

You don’t hire star developers, put them together, and poof get a great team. It’s the other way around. When developers form a great team, the team makes us into great developers.

Part 2: Theory

Why? Why do great teams make great people?

I have a theoretical framework to explain this.

It starts with Gregory Bateson. His father, William Bateson, invented the term genetics and pulled Mendel’s work out of obscurity (with the help of a team of people: his wife and other women, since proper biologists at the time scoffed at this work). Gregory Bateson had a camerata: the Macy Cybernetics Conferences, a series of ten conferences over several years sponsored by the Macy Foundation to advance medical research. Gregory Bateson was a pioneer of cybernetics, now known as Systems Thinking.

Systems thinking is to the present era what the scientific method was for the Renaissance. It is a new way of analyzing problems and experimenting.

Another of his contributions: his daughter, Nora Bateson.

She takes systems thinking a step past where it was in her father’s day. In the 1950s-1970s, scientists put tons of numbers into computers and attempted to model entire ecosystems. Catalogue all the parts, and all the interrelationships between the parts, and we can understand the system and predict outcomes, right?

Not so. And it isn’t for lack of numbers, and details, either. Nora Bateson points out that there’s more to a living system than parts and interrelations: the parts aren’t constant. We grow and learn within the system, so that the parts don’t stay the same and the interrelationships don’t stay the same. She gave a word to this, something deeper than any mechanical or model-able system, a learning system composed of learning parts: symmathesy (sim-MATH-uh-see).

When we look at a living system (a team, say, or a camerata), we can see the parts. People, and our environment (like our desks and computers).

people, and a leaf

Systems thinking says, we are more than the components; we are also the interrelationships. Agile recognizes that these matter.

people and a leaf and strings connecting them

But there’s more! Each member of a team learns every day. our environment learns and adapts too, because we change it.

the people and leaves are colorful now

This is a symmathesy. “Sym” = together, “mathesy” = learning.

We work and grow in a symmathesy. This growth results in great people.

This also explains why it is important to bring our whole selves to work: to build this living system, we need to be alive in it.

Teams Developing Software

There’s something extra special about development teams: software is the most malleable material we’ve ever used in engineering, by thousands of times. There’s nothing else like it, and this changes the meaning of “team.”

A team consists of everyone required for me to be successful, regardless of what the org chart says. Success is running useful software that impacts the world outside my team.

My team includes the people I work with, and also the software itself. And the servers it runs on, the databases it uses, and all the tools that we use to interact with the running software: the code, version control, automated tests, deployment scripts, logging.

The software is part of the symmathesy. We learn from it, from its error messages and logs and the data is collects. It learns from us, because we change it!

As long as we can change it, the software is vitae in our symmathesy. We form a sociotechnical system, people and running code. We build this system from within. No wonder this work is challenging!

Programming wasn’t always this hard. Back when I started, I worked on one program, making the changes someone asked for. For data questions, I asked, “How do we use the database?”

These days, I ask, “which database shall we use?” (Or databases.) And how will we get from here to there? I spend more time on, which work should we do? how will we know it’s useful? My perspective includes our whole software system and team, and surrounding systems too. The scope of the system we can hold in our head is the portion of the system we can change.

Mental models

In order to change our system, we need a mental model of it. Each developer has a mental model of the software we work on. Each developer’s mental model is necessarily incomplete and out of date. And, they’re each different from everyone else’s. (this is called Woods’ Law, if I recall correctly.)

We spend time reconciling our mental models enough to communicate with each other; this is a coherence penalty.

Accurate mental models are incredibly powerful. They let us decide how to change a system, how to know whether it worked, and where to watch for extra consequences. They’re also incredibly hard to form and validate. I’ll call out two reasons in particular for this difficulty.

Line of Representation

A developer can’t look down and see the software running. It is made of dynamic processes running in some digital space that we can’t see into. Until TRON is a thing, we can’t go in there and talk to it. This barrier is called the Line of Representation. (DD Woods et al, STELLA Report)

We can only look at screens and type on keyboards; we use software to observe and control other software. Each of these tools is part of our symmathesy. Grow them, and they can grow the accuracy of our mental models.

I like this diagram because it lets me explain my job at Atomist in one sentence: we make tools to help you move tasks from above the line to below the line. Continuous integration is a step in this direction; Atomist takes it farther.

Downhill Invention, Uphill Analysis

This is counterintuitive: it’s easier to build a system from scratch, constructing the mental model as you go along, than to form an understanding of how an already-built system works.

Downhill Invention (grassy hill), Uphill Analysis (rocky hills)

Valentino Braitenberg illustrates this principle in his book Vehicles. He calls it the Principle of Downhill Invention, Uphill Analysis.

Why are there a thousand JavaScript frameworks out there? because it’s easier to build your own than to gain an understanding of React. Even with hundreds of people contributing to documentation, it’s still more mental effort to form a mental model of an existing system than to construct your own. (I didn’t say it was faster, but less cognitively strenuous.)

Not Invented Here syndrome? laziness. Greenfield development? Of course people like it! it’s much easier than getting your head around legacy code.

When you do have a decent mental model of a system, sharing that with others is hard. You don’t know how much you know. If you’re the purple developer in this picture:

below the line, a software system; above it, a purple person with a mental model of the system. A blue person and a green person with very incomplete models and lots of questions.

then you have a mental model of this system because you built it. The green and blue developers have been assigned to help, but they can’t change the system because they don’t understand it. Meanwhile, you may be changing the system fast enough that it’s impossible for them to get a grasp on it, no matter how smart they are. (I’ve been in their situation.) The solution is to work together on the system, and invest attention in transferring your mental model. Until then, Blue and Green get in your way.

Which developer in this picture is ten times more productive than the others?

Individual vs Group Interests

This brings us to the conflict between advancing your own reputation and contributing to the group.

The race to be first has to be reconciled in science with the need and the norm of sharing.

Following the Camerata’s work, Caccinni argued with Peri and Runiccini and Cavalieri (who was not a member, but corresponded with the Camerata) over who invented the stile rappresentativo, that amazing innovation of one melody plus some accompaniment.

When people are evaluated as individuals, there’s incentive to hold back from sharing. To hoard recognition.

Recognition and esteem accrue to those who have… made genuinely original contributions to the common stock of knowledge.

This is dangerous, because the useful ideas are the ones we contribute. The mental models we hoard make us look good; those we share make the whole team powerful.


Productivity is my personal output. Generativity is the difference between the team’s output with me and without me.

It’s possible to be quite productive personally and have negative generativity, holding the rest of the team back. It’s common to see a person whose own work is slow, but who helps and teaches everyone else. If we start recognizing and crediting generative actions, we build our symmathesy.

The output of everything we do is some side effect on the world and the next version of ourselves. Generativity is about growing the team’s output over time, and each member of the team grows at the same time.


It’s counterintuitive: to become great, put the team first. Aiming first for my own greatness, I limit myself.

Great developers aren’t born.
They aren’t self-made.
Great developers are symmathesized!


Part 3: Predictions

There’s one more piece of the Camerata’s story that surprised me, that gave me a new idea about today’s world.

Surrounding Culture

The Camerata existed in the late Renaissance.

there was a sense of innovation in the air

When the world is ready for an idea, it doesn’t come to just one person.

the Camerata “were like midwives to a sixteenth century which was pregnant with the peculiar conjunction of social, ideological, and cultural ideas and practices from which the opera emerged.”

It was a time of increasing variation; regional and local distinctions emerged, in spite of the Church aiming for uniformity.

the very existence of such groups as social institutions was a product of the time

Right now, the software industry is letting teams try things. Developers are hard to hire, and the work we’re doing is hard, so we get to experiment, even though companies prefer uniformity.

We contribute to the larger world by learning and spreading new practices. Ideas from agile development have spread into other business areas and improved lives.

I’m glad to be alive at a time of cameratas.

Recognition of Art

Somewhere around the beginning of the Renaissance, there was a shift in the way painters, poets, and musicians worked. They used to be tradesmen: they had guilds and apprenticeships. Painters learned how to paint, and musicians how to play the instrument. This guaranteed competence; if you hired someone to paint a scene, they could paint something reasonable. It didn’t guarantee talent; guild membership was more about whose kid you were.

During the Renaissance these guilds lost power. Competence was harder to guarantee, but individual talent was recognized. Painters and poets specialized in subject matter. People noticed that there was some common factor to music, painting, poetry, drama — some indefinable essence that was more than competency with a brush.

Some spark. some Art!

Before, as tradesmen, painters hung out with painters, sculptors with bronzeworkers etc. After Art became a thing, artists studied in academies, and they talked to intellectuals of many kinds.

transformation of homogeneous artistic circles into “cultured” circles: poets, artists, amateurs, and laymen alike.

It reminds me of a move from typing Java in a cubicle into an agile team, with discussions including designers and testers and businesspeople. I’m no longer painting in acrylics. I’m painting something.

Software is not a craft.

We aren’t housepainters. Competence is not supreme in a shifting, complex system. Our users’ needs change, our technologies change, and every change we undertake is one we’ve never made before. We cannot possess all the skills we need; our secret weapon is learning, never perfection.

Software is not an art.

Neither are we creating a portrait or a poem, where there is such a thing as “done.” What we build affects people more directly than works of art. To do this it must keep running, it must work on their devices, it must keep changing. Software has a greater power to change people by changing the world, but these systems are more than one person can make, and they never complete.

Serious software development is the practice of symmathesy.

Software is not Art. Software is the next thing after art. It is something new, that we’ve never been able to do before.

And that implies that our current time is the next thing after the Renaissance. We are developing whole new ways of being human together. I am thrilled to be a part of this, and to be in the thick of it. To be more than a craftsperson, more even than a developer.

I am a Symmathecist, in the medium of software. (sim-MATH-uh-sist)

“Jessica Kerr, Symmathecist in the medium of software”

I want to practice symmathesy, to grow as part of a growing system, a system with people of many different skills. I want to care for and change software that changes the world outside my symmathesy, even as it changes me.

The only problem with this idea is that it has my name on it. Ideas are only valuable when they’re shared, spread, contributed to the common knowledge. Every person who spreads an idea is part of it. So join me, spread this.

Do you have a camerata? Can you help build one?

How do you practice symmathesy, consciously? How can we educate ourselves and each other in this practice? Are you a symmathecist, too?

the learning-people-and-interrelationships picture, with “Symmathecist, in the medium of software.”

The camerata changed history; we feel their impact every day. (Sample: my daughter singing Hamilton, the modern music-drama.)

So will we.

Does caring scale?

Does caring scale?

We can’t all care about everything. We have cities without clean water, a criminal justice system full of injustice, and oceans clogged with paper bags. There are too many important causes to catalogue, much less champion. We can only contribute to a few.

We can’t all care about everything. At my house, I care that leftover food is sealed in containers and that the trash can has a bin liner. Eric cares that the counter is clean. Ivy cares that we eat the food before it expires… but she’ll put it in the fridge covered halfheartedly with Cling wrap! when that container has a lid!!

I am filled with righteous fury at the unsealed tupperware. After cursing, I calm down and remember this is not, in fact, a moral issue. We can’t get everything right; this is just the one that I care about.

If I want change, the thing is not to exhort my household to respect the sanctity of lids. They aren’t anti-lid. The thing to do is: make the lid more convenient than Cling-wrap.

See, I used to store my containers separately from their lids. This is efficient for storage, because the containers nest. But! Sealed lids are more important than space efficiency! Other members of the household did not have the lid-container correspondence memorized.

I changed the system. Now we store containers lid-on. Fewer containers, but all of them useful.

The results have been positive; lid usage has increased in my household.

We can’t all care about everything. At work, Clay cares about test coverage. I care about traceability. Rod cares about consistent, clear naming. All important, all a balance when time is limited.

Rod also cares about capitalization of issue titles. And he wants every issue title to have a description.

He tried complaining about my sloppy issues, but I was proud of myself for recording the bug at all. He could have invoked authority — he is the CEO — but instead he added automation. He coded two NewIssueReactions into our software delivery machine. One capitalizes the title of every new issue. It doesn’t complain, it just fixes it. The description is harder, so that reaction sends me a Slack DM: please add a description. A link to the issue makes it easy for me.

We now have 100% issue title capitalization and a higher description rate than before.

Coraline Ehmke wanted good codes of conduct for open source projects, so she made it easier for people.

If you want less litter in the world, put recycle bins around.

People care, but we can’t all care about everything. When you care a lot, work at making it easy for other people to act constructively. A big effort from you and make a small effort from everyone can make a serious impact.

Folding Tradeoff Space

automation changes the rules we play by

Fast v Safe

Tradeoffs are a fact of life. Choose between safety and speed, stability and flexibility. More of one sacrifices the other.

fast or safe, we are somewhere on that spectrum

I remember a time when this described our decisions about software deployments. Deployments caused problems, so we did fewer of them.

Then came Continuous Integration and Continuous Deployment. With a lot of work, we get automated tests and monitoring, automated deployment and rollback. Then we learned that small deployments are safer — so much safer that many fast deployments added up to less danger than one bulk deployment.

Fast and Safe

It’s like, this automation plus culture change has folded space — these arrows now point in the same direction.

fast and safe point in the same direction, they are no longer in conflict

Bending tradeoff space like this wasn’t free.

Tests double our application code, and infrastructure adds even more code. Harder: we had to change the culture. Think in smallest shippable units. Alleviate the next bottleneck after deploys aren’t it, and the next.

Without weeks of manual testing, we had to create new ways to answer “Does it work?” Without ops people standing by, we needed new ways to answer “is it up?” When each person makes deployment decisions individually, they need tools to support the decision with visibility, and tools to implement the decision with fast rollback in case of surprise.

This automation is expensive, but we wouldn’t go back to the old days of dangerous, slow deployments.

Flexible v Consistent

I want to talk about another tradeoff space, this one in “Do we create custom internal software?” As a business, I want internal software to increase the consistency, legibility, and speed of my employees’ work. We think of the cost of software in terms of money and time, work to create it and to keep it happy. There’s a deeper cost to the organization, though: ossification.

spectrum of flexible or consistent

The consistency enforced by software is great for managers and administrators; they get legibility, meaning they can track work at a high level. These are the people making decisions about software requirements.

I once worked on workflow tracking software for a research organization. The heads of the development effort bragged that they’d encoded the knowledge of the scientists, so they could leave and we’d still have it. The research could proceed… in exactly the same way. Any substantial change took (this is extreme, but it was ten years ago) eighteen months to roll out. So much for scientific innovation! oh but it was great for the higher-ups tracking progress of each experiment as it moved through this regimented system.

This consistency is not optimal for learning, not for daily patient care or customer service. People are adaptable, but software inhibits this adaptation. Ask any nurse, social worker, or hotel receptionist how they feel about the software they have to use. (Ask them when they are off duty.) They’ll tell you about frustrations, time wasted, obstacles to doing great work.

Compared to a paper form, an electronic form is constraining. Compared to talking to each other, coordinating via task trackers is limiting. Software ossifies processes. It impedes learning and improvisation.

Flexible and Consistent

How could we fold this tradeoff space?

This tradeoff exists when the people using the software can’t change the software. Maybe it can help to look at one place where this tradeoff does not exist, where it is naturally folded.

flexible and consistent in the same direction: local automation

When the users are also the system movers, calcification doesn’t happen.

Software teams developing for themselves gain consistency, legibility, speed and more — while adding flexibility. When deployment is automated in one place, we can improve the process without the work of disseminating information.

Automation helps with testing. It helps with deployment. What else can we automate for ourselves? feature tracking, dependency updates, information gathering, project creation, common code change tasks?

Automation isn’t cheap, but it is priceless. Let’s smooth our own work with comprehensive automation. We might even get better at moving systems and help the whole business flex.

Shaping our Tools to the Flow of Conversation: Issue Creation

Collaboration is hard. There are the essential difficulties of dealing with people, and then there are incidental challenges in the tools we use. Communicating with people, while clicking into to Travis to see the build status and asking: where is this deployed? Who tested it? I want to remove all those incidental challenges, even the tiny ones.

We collaborate with tools: Slack, an issue tracker, source control, many more. We want the best tool for each purpose, but switching between them is extra work for my brain. A tiny bit of extra work, but every distraction matters when I’m focused on a complex task.

For instance, we’re talking in Slack, and realize we need to track this with a GitHub issue. Sometimes two of us create an issue at the same time — oops! I want to create that issue right in the conversation where people can see it.

The Atomist bot can do this for me:

@atomist create issue

I don’t need all the functionality of GitHub Issues in Slack, just enough plus a link. This way I rarely have to switch contexts, and neither does anyone else there in the conversation. I can claim the issue and vote for it and add a label right here. If I pop over to GitHub and update the issue, the Slack message redraws when new information has come to light.

This is a small thing.

Small things matter when they let us keep more of our brain on the hard problems of coding and collaboration with people. Every tiny mental bump I can smooth out for my team makes each of us effectively smarter.

Create Issue is a tiny piece of Atomist’s built-in automation, interweaving the tools we have to suit the way we smoothly work. There’s more: commit messages are pretty and are integrated with build and deploy status, PR notifications are mergeable, even new projects are created in line with the conversation.

We don’t need a new interface for collaboration. We need to make the ones we have as perfect as possible, then customize per team.

As developers, we have the power to create the world we want: the world that gets the tiny problems out of our way so we can solve the hard stuff.

Spend some time automating your own work, with whatever tools you have. Invite the Atomist bot into your team to get Create Issue and other integrations. Join the development automation conversation in our community Slack. We’d love to hear how you shape your tools to fit your flow.

the sweet spot: Local Automation

When you learn to code, you acquire a superpower: automation. We turn the computer into a machine to do our (very specific) bidding. We get paid to automate what other people want. We can and should use this superpower for ourselves too!

Levels of automation

Are we programmers or software developers?

Probably both.

I think of a program as a piece of code that runs on my box and does something useful for me. It can be a one-line shell script or a Scala program backed by a graph database that downloads all my dependencies.

Software is also made of code. Lots of code, code on servers and code in browsers and code on mobile devices. Software runs on all the computers and is useful to millions of people. It has compatibility, integration, redundancy, persistence, security, tests, user experience, multi-region back ends and multi-platform front ends. It has program managers and technical writers and designers and operations. It has stakeholders.

Scaling and safety and compatibility make software super expensive: thousands or millions of times more expensive than a program. And then there’s the risk: will the Software be useful? (If it’s internal software people are required to use, it might even make their jobs harder. Net negative benefit.) What changes will make it more useful?


Software and programs are degrees of automation. We build this automation in order to make a system work more smoothly. Optimizing that system is easy when the beneficiary of a change is the same person who can make it — in a program, I am my stakeholder, I define “useful.” In Software, Optimizing is super hard because the people benefiting from the software and the people who can change it are distant in the communication graph. The feedback loop is long and circuitous.

On one hand, we can work on bringing developers closer to people who can measure benefits. On the other hand, can we find a place where this is already the case?

In development teams! Developers automating for their own team is low effort, high payoff. Personally, I can make programs that only benefit me, which is limited. Go one level up: how much more work is it to write code that benefits my team? If we automate what we already do, then: it only has to run on a few known computers, no one is forced to use it, everyone who uses it can change it so tests are optional. This gives me some scale of value, with little scale in cost. If it turns out really useful, other teams in the org can copy it.

I wish I had automation to set up a build on Travis. A minimum value is certain: I want it right now. The cost is uncertain, but discoverable. I can risk spending one 20-minute iteration on it. At the end of that iteration, I’ll have more information about the cost. I can give it another 20 minutes, or set the idea aside until I hit the same problem again, which gives me information about the value. My pair and I (or the mob in mob programming) become a microcosm of a product team, for whatever timebox we set.

It helps, so do it more.

At some point we automated tests, installed version control, set up continuous integration. These are automations we no longer do without, although at one point we didn’t know how useful they’d be. What will be the next automation we won’t work without?

Here are some examples of local or personal automation that I heard about or used today.

As developers we have this superpower of automation. I think we should use it more on ourselves. We will learn more about our own work by studying how to improve it. And we will improve it, and continually grow our ability to create reliable, persistent, user-experienceable, compatible Software.

Write shell scripts! and Slack bots! and Jenkins plugins! At Atomist I have a platform to more easily scale up my automations from local to team; we’re building it. Don’t wait for us, though. Local automation is a sweet spot no matter what tool you use. Accelerate yourself.