the Origins of Opera and the Future of Programming

Or: Collective problem solving in music, art, science, and software.

(video, or TL;DR)

At the end of this post is an audacious idea about the present and future of software development. In the middle are points about mental models: how important and how difficult they are. But first, a story of the origins of Opera.

Part 1: Examples

The Camerata

The Florentine Camerata were a group of people who met in Florence in the 16th century. They had a huge impact on history and on each other.

(Caveat: I’m not a student of music or music history. I listened to this Great Course and heard about this Camerata and noticed a resemblance to some teams I’ve seen and been in. So I looked it up further, and came upon Collective “Problem-Solving” in the History of Music: the Case of the Camerata by Dr. Ruth Katz in the Journal of the History of Ideas, and toward the end of it my mind was blown. More on that later.)

Camerata literally means a small orchestra or choir. This Camerata was a diverse group of people who gathered and worked on a common problem: they were bored with polyphony, the esteemed music of their day. (Sample: Palestrina) Polyphony is very pretty: it has around four melodies, each of equal importance. Each has a logic of its own, rather than melody and accompaniment. Polyphony is intellectually rewarding, but you need technical understanding to appreciate it fully. What feeling it conveys comes through auditory qualities.

The Camerata asked the revolutionary question: what if you could understand the words?

Methods

The Camerata included (all quotes in this style are from Katz)

musicians, artists, poets, astrologers, philosophers, scientists
who met informally under the aegis of Bardi and Corsi.

People with diverse skills and perspectives worked together. They had sponsorship; Giovanni d’Bardi was Count of Venoa, and loved to surround himself with interesting people.

Their aim was to reform the polyphonic music of the day
and they believed that the best way to do so was to renovate the ancient Greek practice of setting words to music

They shared a common goal; they were unsatisfied with what Vincenzo Galilei (member of the Camerata, father of Galileo) called “that corrupt and incomprehensible contemporary music.”

And they had a common strategy. They didn’t really know what the Greeks did, but this lent legitimacy to their ideas. Like citing computer science papers from the 70s.

Their real high-level objective was horizonal, and more specific than moving away from polyphony:

Their principal aim was to find the optimum formula for wedding words and music.

Here, “optimum” is measured as “maximally effective in communicating… the specific meanings and emotions appropriate to the text.”

Qualities

The Camerata talked a lot, and listened to each other talk.

“I learned more from their learned discussions than I learned from descant in over thirty years” — Caccini, renowned singer

(I had to look up “descant.” It means long-winded discourse. Like you’re experiencing now, sorry.)

But they weren’t all talk.

the Camerata constituted not only a forum for theoretical discussions, but also a workshop, a “laboratory” for the creation and performance of music.

They practiced together! And experimented. The beginning of the scientific method is a big part of the Renaissance, and it intertwines with art. We have a new way of thinking, ways of asking the universe questions. Vincenzo Galilei varied lengths and tensions of strings and found the ratios that make chords.

The Camerata didn’t always get along. There was rivalry between Bardi and Corsi, the two chief sponsors. Bardi preferred talking, Corsi wanted to play more music. These feed each other. There was disagreement over style between Caccini and Peri, the two musical stars. Peri wanted to focus on the words, with a bit of music; Caccini wanted the singing to stand out, while also understanding the words. These tensions lead to a balance.

They did code review!

presentations made… were commented on formally by “defenders” and “censors” who were nominated for the occasion.

I like that criticism came from people designated to the role, not the asshole who doesn’t like you and takes it out on your code. (Technically, this took place in the Alterati, another meetup with a lot of the same people.)

Outcomes

Over the years, this team changed history. They invented the music-drama, and a style of music that conveyed more meaning. (Sample: Monteverdi, a first composer to adopt the Camerata’s style. If you know Italian, you can probably understand the words.)

What about the individuals? Their outcomes were exceptional too! Here are some of their publications:


As composers of operas and authors of scientific treatises, these half-dozen people are fewer than half of the Camerata members who have Wikipedia articles. Really, what are the chances, if you’re alive in the sixteenth century, that you have a Wikipedia article today? These people did pretty well for themselves. They grew out of the Camerata.

Also in Science

This pattern of a group of people coming together to solve a problem is not unique to music — it’s the common case.

the Camerata resembles the kind of “invisible college” which is the key to creativity in science.

This “invisible college” is an association of people who share ideas. Who build a new reality together, then spread it to advance the wider culture.

We like to give the Nobel Prize to one or two people. But who worked in their lab? Who did they correspond with?

When Jon Von Neumann went to Los Alamos for the Manhattan Project, so did two or three mathematicians that he went to high school with. Really, you grow up in Hungary, what are your chances of getting to Los Alamos? They built each other up.

These invisible colleges share:

tacit understandings concerning appropriate methods of research

(processes and values)

priority problems

(this means they fight over who was first; more on this later)

and the shorthand communication which shared work implies.

We can move quickly together because we share common ground, compatible mental models. This is super fun, when I get to this point with my team.

Also in Art

People work together to develop their individual styles. Usually in Paris, it seems.

the salon, the coffeehouse, the café as breeding places of artistic creativity

In the nineteenth century, a group of artists broke from the mainstream and developed Impressionism.

coping with a common puzzle which they, separately and as a group, tried to solve

Van Gogh lived in the Montmartre district with the other artists and dealers and critics. When I visited his museum in Amsterdam, my favorite part was all the paintings by his friends and associates; they developed each other as painters.


One of these (my personal favorite) was Paul Gauguin, the one Van Gogh cut his ear off over. Gauguin went on to influence Picasso.

Picasso was at the center of many social circles in Paris over the decades. Writers, photographers, philosophers.


One painter who dipped in and out of his camerata was Aleksandra Ekster, who took the ideas of Cubism back to Kiev, where her studio was its own place of idea exchange.


One of her high school classmates and friends was Alexander Bogomazov, and a print of his lives on my bedroom wall.

my own little museum. includes “Head” by Bogomazov next to a piece by my daughter

This brings us to the modern day, where we can find examples of this phenomenon in software teams.

Also in Software

One camerata emerged in London around 2003–2006. This group gave us Continuous Integration, Continuous Delivery, DevOps. Many of these people worked at ThoughtWorks. They weren’t all on the same project, but they talked to each other, and they solved the problem of: Does deployment have to be so hard?

Jez and Dan and Chris Read produced The Deployment Production Line. Later, Dan went to invent BDD, Sam Newman became a prophet of microservices, and more. I keep meeting conference speakers who were part of this group.


Another example: the early Spring team, around the same time. They came together online, from all around the world, to solve the problem of: do we really have to use J2EE? and made Java development less painful for everyone. Today, Java development is (approximately) everywhere, and Spring is everywhere Java is.

That group of developers and businesspeople produced an inordinate number of founders and CEOs and partners.

Rod Johnson, CEO Atomist; Christian Dupuis VP Eng, Atomist; Steven Schuurman, Founding CEO, Elastic; Neelan Choksi, President and COO, Tasktop; Rob Harrop, CEO, Skipjaq; Adrian Colyer, Venture Partner, Accel

Personally, I’ve been part of three teams that grew me as a developer and as a person. The tech meetup scene in St Louis is a source of growth too. We have several groups of about twenty, and a lot of overlap so we can talk to the same people multiple times a month about interesting things.

In all of these examples, we can observe a phenomenon:

Great Teams Make Great People.

You don’t hire star developers, put them together, and poof get a great team. It’s the other way around. When developers form a great team, the team makes us into great developers.

Part 2: Theory

Why? Why do great teams make great people?

I have a theoretical framework to explain this.

It starts with Gregory Bateson. His father, William Bateson, invented the term genetics and pulled Mendel’s work out of obscurity (with the help of a team of people: his wife and other women, since proper biologists at the time scoffed at this work). Gregory Bateson had a camerata: the Macy Cybernetics Conferences, a series of ten conferences over several years sponsored by the Macy Foundation to advance medical research. Gregory Bateson was a pioneer of cybernetics, now known as Systems Thinking.


Systems thinking is to the present era what the scientific method was for the Renaissance. It is a new way of analyzing problems and experimenting.

Another of his contributions: his daughter, Nora Bateson.


She takes systems thinking a step past where it was in her father’s day. In the 1950s-1970s, scientists put tons of numbers into computers and attempted to model entire ecosystems. Catalogue all the parts, and all the interrelationships between the parts, and we can understand the system and predict outcomes, right?

Not so. And it isn’t for lack of numbers, and details, either. Nora Bateson points out that there’s more to a living system than parts and interrelations: the parts aren’t constant. We grow and learn within the system, so that the parts don’t stay the same and the interrelationships don’t stay the same. She gave a word to this, something deeper than any mechanical or model-able system, a learning system composed of learning parts: symmathesy (sim-MATH-uh-see).

When we look at a living system (a team, say, or a camerata), we can see the parts. People, and our environment (like our desks and computers).

people, and a leaf

Systems thinking says, we are more than the components; we are also the interrelationships. Agile recognizes that these matter.

people and a leaf and strings connecting them

But there’s more! Each member of a team learns every day. our environment learns and adapts too, because we change it.

the people and leaves are colorful now

This is a symmathesy. “Sym” = together, “mathesy” = learning.

We work and grow in a symmathesy. This growth results in great people.

This also explains why it is important to bring our whole selves to work: to build this living system, we need to be alive in it.

Teams Developing Software

There’s something extra special about development teams: software is the most malleable material we’ve ever used in engineering, by thousands of times. There’s nothing else like it, and this changes the meaning of “team.”


A team consists of everyone required for me to be successful, regardless of what the org chart says. Success is running useful software that impacts the world outside my team.

My team includes the people I work with, and also the software itself. And the servers it runs on, the databases it uses, and all the tools that we use to interact with the running software: the code, version control, automated tests, deployment scripts, logging.

The software is part of the symmathesy. We learn from it, from its error messages and logs and the data is collects. It learns from us, because we change it!

As long as we can change it, the software is vitae in our symmathesy. We form a sociotechnical system, people and running code. We build this system from within. No wonder this work is challenging!

Programming wasn’t always this hard. Back when I started, I worked on one program, making the changes someone asked for. For data questions, I asked, “How do we use the database?”


These days, I ask, “which database shall we use?” (Or databases.) And how will we get from here to there? I spend more time on, which work should we do? how will we know it’s useful? My perspective includes our whole software system and team, and surrounding systems too. The scope of the system we can hold in our head is the portion of the system we can change.


Mental models

In order to change our system, we need a mental model of it. Each developer has a mental model of the software we work on. Each developer’s mental model is necessarily incomplete and out of date. And, they’re each different from everyone else’s. (this is called Woods’ Law, if I recall correctly.)


We spend time reconciling our mental models enough to communicate with each other; this is a coherence penalty.

Accurate mental models are incredibly powerful. They let us decide how to change a system, how to know whether it worked, and where to watch for extra consequences. They’re also incredibly hard to form and validate. I’ll call out two reasons in particular for this difficulty.

Line of Representation

A developer can’t look down and see the software running. It is made of dynamic processes running in some digital space that we can’t see into. Until TRON is a thing, we can’t go in there and talk to it. This barrier is called the Line of Representation. (DD Woods et al, STELLA Report)


We can only look at screens and type on keyboards; we use software to observe and control other software. Each of these tools is part of our symmathesy. Grow them, and they can grow the accuracy of our mental models.

I like this diagram because it lets me explain my job at Atomist in one sentence: we make tools to help you move tasks from above the line to below the line. Continuous integration is a step in this direction; Atomist takes it farther.

Downhill Invention, Uphill Analysis

This is counterintuitive: it’s easier to build a system from scratch, constructing the mental model as you go along, than to form an understanding of how an already-built system works.

Downhill Invention (grassy hill), Uphill Analysis (rocky hills)

Valentino Braitenberg illustrates this principle in his book Vehicles. He calls it the Principle of Downhill Invention, Uphill Analysis.

Why are there a thousand JavaScript frameworks out there? because it’s easier to build your own than to gain an understanding of React. Even with hundreds of people contributing to documentation, it’s still more mental effort to form a mental model of an existing system than to construct your own. (I didn’t say it was faster, but less cognitively strenuous.)

Not Invented Here syndrome? laziness. Greenfield development? Of course people like it! it’s much easier than getting your head around legacy code.

When you do have a decent mental model of a system, sharing that with others is hard. You don’t know how much you know. If you’re the purple developer in this picture:

below the line, a software system; above it, a purple person with a mental model of the system. A blue person and a green person with very incomplete models and lots of questions.

then you have a mental model of this system because you built it. The green and blue developers have been assigned to help, but they can’t change the system because they don’t understand it. Meanwhile, you may be changing the system fast enough that it’s impossible for them to get a grasp on it, no matter how smart they are. (I’ve been in their situation.) The solution is to work together on the system, and invest attention in transferring your mental model. Until then, Blue and Green get in your way.

Which developer in this picture is ten times more productive than the others?

Individual vs Group Interests

This brings us to the conflict between advancing your own reputation and contributing to the group.

The race to be first has to be reconciled in science with the need and the norm of sharing.

Following the Camerata’s work, Caccinni argued with Peri and Runiccini and Cavalieri (who was not a member, but corresponded with the Camerata) over who invented the stile rappresentativo, that amazing innovation of one melody plus some accompaniment.

When people are evaluated as individuals, there’s incentive to hold back from sharing. To hoard recognition.

Recognition and esteem accrue to those who have… made genuinely original contributions to the common stock of knowledge.

This is dangerous, because the useful ideas are the ones we contribute. The mental models we hoard make us look good; those we share make the whole team powerful.

Generativity

Productivity is my personal output. Generativity is the difference between the team’s output with me and without me.

It’s possible to be quite productive personally and have negative generativity, holding the rest of the team back. It’s common to see a person whose own work is slow, but who helps and teaches everyone else. If we start recognizing and crediting generative actions, we build our symmathesy.

The output of everything we do is some side effect on the world and the next version of ourselves. Generativity is about growing the team’s output over time, and each member of the team grows at the same time.

Obliquity

It’s counterintuitive: to become great, put the team first. Aiming first for my own greatness, I limit myself.

Great developers aren’t born.
They aren’t self-made.
Great developers are symmathesized!

(sim-MATH-uh-sized)

Part 3: Predictions

There’s one more piece of the Camerata’s story that surprised me, that gave me a new idea about today’s world.

Surrounding Culture

The Camerata existed in the late Renaissance.

there was a sense of innovation in the air

When the world is ready for an idea, it doesn’t come to just one person.

the Camerata “were like midwives to a sixteenth century which was pregnant with the peculiar conjunction of social, ideological, and cultural ideas and practices from which the opera emerged.”

It was a time of increasing variation; regional and local distinctions emerged, in spite of the Church aiming for uniformity.

the very existence of such groups as social institutions was a product of the time

Right now, the software industry is letting teams try things. Developers are hard to hire, and the work we’re doing is hard, so we get to experiment, even though companies prefer uniformity.

We contribute to the larger world by learning and spreading new practices. Ideas from agile development have spread into other business areas and improved lives.

I’m glad to be alive at a time of cameratas.

Recognition of Art

Somewhere around the beginning of the Renaissance, there was a shift in the way painters, poets, and musicians worked. They used to be tradesmen: they had guilds and apprenticeships. Painters learned how to paint, and musicians how to play the instrument. This guaranteed competence; if you hired someone to paint a scene, they could paint something reasonable. It didn’t guarantee talent; guild membership was more about whose kid you were.

During the Renaissance these guilds lost power. Competence was harder to guarantee, but individual talent was recognized. Painters and poets specialized in subject matter. People noticed that there was some common factor to music, painting, poetry, drama — some indefinable essence that was more than competency with a brush.

Some spark. some Art!

Before, as tradesmen, painters hung out with painters, sculptors with bronzeworkers etc. After Art became a thing, artists studied in academies, and they talked to intellectuals of many kinds.

transformation of homogeneous artistic circles into “cultured” circles: poets, artists, amateurs, and laymen alike.

It reminds me of a move from typing Java in a cubicle into an agile team, with discussions including designers and testers and businesspeople. I’m no longer painting in acrylics. I’m painting something.

Software is not a craft.

We aren’t housepainters. Competence is not supreme in a shifting, complex system. Our users’ needs change, our technologies change, and every change we undertake is one we’ve never made before. We cannot possess all the skills we need; our secret weapon is learning, never perfection.

Software is not an art.

Neither are we creating a portrait or a poem, where there is such a thing as “done.” What we build affects people more directly than works of art. To do this it must keep running, it must work on their devices, it must keep changing. Software has a greater power to change people by changing the world, but these systems are more than one person can make, and they never complete.

Serious software development is the practice of symmathesy.

Software is not Art. Software is the next thing after art. It is something new, that we’ve never been able to do before.

And that implies that our current time is the next thing after the Renaissance. We are developing whole new ways of being human together. I am thrilled to be a part of this, and to be in the thick of it. To be more than a craftsperson, more even than a developer.

I am a Symmathecist, in the medium of software. (sim-MATH-uh-sist)

“Jessica Kerr, Symmathecist in the medium of software”

I want to practice symmathesy, to grow as part of a growing system, a system with people of many different skills. I want to care for and change software that changes the world outside my symmathesy, even as it changes me.

The only problem with this idea is that it has my name on it. Ideas are only valuable when they’re shared, spread, contributed to the common knowledge. Every person who spreads an idea is part of it. So join me, spread this.

Do you have a camerata? Can you help build one?

How do you practice symmathesy, consciously? How can we educate ourselves and each other in this practice? Are you a symmathecist, too?

the learning-people-and-interrelationships picture, with “Symmathecist, in the medium of software.”

The camerata changed history; we feel their impact every day. (Sample: my daughter singing Hamilton, the modern music-drama.)

So will we.

Does caring scale?

Does caring scale?

We can’t all care about everything. We have cities without clean water, a criminal justice system full of injustice, and oceans clogged with paper bags. There are too many important causes to catalogue, much less champion. We can only contribute to a few.

We can’t all care about everything. At my house, I care that leftover food is sealed in containers and that the trash can has a bin liner. Eric cares that the counter is clean. Ivy cares that we eat the food before it expires… but she’ll put it in the fridge covered halfheartedly with Cling wrap! when that container has a lid!!

I am filled with righteous fury at the unsealed tupperware. After cursing, I calm down and remember this is not, in fact, a moral issue. We can’t get everything right; this is just the one that I care about.

If I want change, the thing is not to exhort my household to respect the sanctity of lids. They aren’t anti-lid. The thing to do is: make the lid more convenient than Cling-wrap.

See, I used to store my containers separately from their lids. This is efficient for storage, because the containers nest. But! Sealed lids are more important than space efficiency! Other members of the household did not have the lid-container correspondence memorized.

I changed the system. Now we store containers lid-on. Fewer containers, but all of them useful.


The results have been positive; lid usage has increased in my household.

We can’t all care about everything. At work, Clay cares about test coverage. I care about traceability. Rod cares about consistent, clear naming. All important, all a balance when time is limited.

Rod also cares about capitalization of issue titles. And he wants every issue title to have a description.

He tried complaining about my sloppy issues, but I was proud of myself for recording the bug at all. He could have invoked authority — he is the CEO — but instead he added automation. He coded two NewIssueReactions into our software delivery machine. One capitalizes the title of every new issue. It doesn’t complain, it just fixes it. The description is harder, so that reaction sends me a Slack DM: please add a description. A link to the issue makes it easy for me.


We now have 100% issue title capitalization and a higher description rate than before.

Coraline Ehmke wanted good codes of conduct for open source projects, so she made it easier for people.

If you want less litter in the world, put recycle bins around.

People care, but we can’t all care about everything. When you care a lot, work at making it easy for other people to act constructively. A big effort from you and make a small effort from everyone can make a serious impact.

Understanding code under change

Understanding code under change

Why is it so hard to jump into a team and start contributing to code? Why is someone else’s code so darn incomprehensible?

One reason is that the code is a snapshot of a system in process. If you know where it was and where it’s aiming, the middle makes more sense.

For instance, I’m working on a system that lists steps between code push and deployment, then executes them. We started out using GitHub statuses for this, as the simplest thing that could work.


Now that we are sure this is useful, we want them to be their own thing, a Goal in our cortex (graph database). We’ve started by creating a custom cortex Goal event, and creating Goals for every push. We copy those to GitHub statuses, and the rest of the program works off those. Incrementally, I’m changing code to use Goals instead of Statuses. If you came into the code at this point without that background knowledge, you’d find it totally confusing. Why do some things update Goals and others Statuses? How do those even connect? What kind of loon wrote this convoluted mess?

If you’re the person who has been around long enough to know where the code was, and if you understand the desired architecture it’s moving to, be kind to people without that context.

If you’re the person reading ridiculous code, seek out someone with background and ask them. There’s probably a story about this code, both from its past and its future. Then you can contribute to moving toward that future, and eventually to closure.

As soon as everything is saving Goals and triggering off Goals, then it’ll copy only a few to GitHub statuses for visibility, and I’ll delete the duplicitous code. I’m off to work on that now before something else super urgent pulls me away and the code gets stuck in this icky swamp forever. (Been there?)

the sweet spot: Local Automation

When you learn to code, you acquire a superpower: automation. We turn the computer into a machine to do our (very specific) bidding. We get paid to automate what other people want. We can and should use this superpower for ourselves too!

Levels of automation

Are we programmers or software developers?

Probably both.

I think of a program as a piece of code that runs on my box and does something useful for me. It can be a one-line shell script or a Scala program backed by a graph database that downloads all my dependencies.

Software is also made of code. Lots of code, code on servers and code in browsers and code on mobile devices. Software runs on all the computers and is useful to millions of people. It has compatibility, integration, redundancy, persistence, security, tests, user experience, multi-region back ends and multi-platform front ends. It has program managers and technical writers and designers and operations. It has stakeholders.

Scaling and safety and compatibility make software super expensive: thousands or millions of times more expensive than a program. And then there’s the risk: will the Software be useful? (If it’s internal software people are required to use, it might even make their jobs harder. Net negative benefit.) What changes will make it more useful?

Optimization

Software and programs are degrees of automation. We build this automation in order to make a system work more smoothly. Optimizing that system is easy when the beneficiary of a change is the same person who can make it — in a program, I am my stakeholder, I define “useful.” In Software, Optimizing is super hard because the people benefiting from the software and the people who can change it are distant in the communication graph. The feedback loop is long and circuitous.

On one hand, we can work on bringing developers closer to people who can measure benefits. On the other hand, can we find a place where this is already the case?

In development teams! Developers automating for their own team is low effort, high payoff. Personally, I can make programs that only benefit me, which is limited. Go one level up: how much more work is it to write code that benefits my team? If we automate what we already do, then: it only has to run on a few known computers, no one is forced to use it, everyone who uses it can change it so tests are optional. This gives me some scale of value, with little scale in cost. If it turns out really useful, other teams in the org can copy it.


I wish I had automation to set up a build on Travis. A minimum value is certain: I want it right now. The cost is uncertain, but discoverable. I can risk spending one 20-minute iteration on it. At the end of that iteration, I’ll have more information about the cost. I can give it another 20 minutes, or set the idea aside until I hit the same problem again, which gives me information about the value. My pair and I (or the mob in mob programming) become a microcosm of a product team, for whatever timebox we set.

It helps, so do it more.

At some point we automated tests, installed version control, set up continuous integration. These are automations we no longer do without, although at one point we didn’t know how useful they’d be. What will be the next automation we won’t work without?

Here are some examples of local or personal automation that I heard about or used today.

As developers we have this superpower of automation. I think we should use it more on ourselves. We will learn more about our own work by studying how to improve it. And we will improve it, and continually grow our ability to create reliable, persistent, user-experienceable, compatible Software.

Write shell scripts! and Slack bots! and Jenkins plugins! At Atomist I have a platform to more easily scale up my automations from local to team; we’re building it. Don’t wait for us, though. Local automation is a sweet spot no matter what tool you use. Accelerate yourself.

The Golden Yak

(This post continues from the Royal Yak, and concludes the series A Taxonomy of Yaks.)

When we improve how we work, we make tasks faster. We make progress smoother. This is magnified when we improve how all our team members work, or our whole community. Now and then, though, an improvement turns into something more: a real change. This is what I’m aiming for when I shave a Golden Yak.

Golden Yak

The Golden Yak is the yak that changes everything, the one that alters your reality. Sometimes when you follow a problem all the way down — how do I make this easier? … now how do I make that easier? … now what if this were no work at all? you find a different universe. Making something sufficiently easier can produce a sea change, which changes our behavior.

quotes from yakbreeder.com

Example: git. It changed my behavior. Git gives me logs at my fingertips, where they used to be behind a network connection so I never used them. And it gives me a capability I didn’t have before: separating saving my work from sharing my work. The combination is magic; now I save my work carefully with a story and later read that story.

Continuous Integration. This was a huge yak for the people who first did it. It’s still a lot of work, but we wouldn’t live without it, because we (as an industry) have learned that it smooths our future work. It changes our universe by eliminating the tradeoff between frequent vs safe releases. Automated tests also change our work. They reduce fear, and that’s huge. Tests turn canyons into potholes.

Golden yaks change behavior. They start by making something so much easier that we do it way more, then they give us an option we didn’t have before or eliminate a tradeoff.

Here’s a tiny one: when I find myself typing the same command a lot, there’s the question: create an alias for this? well, I have an alias set up for “open .bash_profile in an editor” and another for “reload the aliases in .bash_profile”. Creating an alias is so easy now, there’s no reason not to. It’s gone from a tradeoff (is it worth it?) to a “duh.” Like making a commit. Don’t even ask, just do it!

Here’s a team-sized one: automate spinning up a new project. This takes away a tradeoff: “this is really a separate thing, I should make a service for it… but it takes so long, I should glom it on to an existing one.” With the Atomist bot, I made it two slack commands and two in-Slack button clicks to create a repo and a channel and a build and a deployment, and bam I have an empty service in production ready for my new endpoint. It makes creating a new service “duh.” That decision is now all about architecture, not about tradeoffs. This yak is shaved once and for all. (until there’s a change in infrastructure. And then I’ll accommodate that change in one place, instead of various documentations and people’s heads.)

Here’s one that might turn out bigger: Dependency management, unsolved problem. Code reuse, not all it’s cracked up to be. I am against creating dependencies for code that happens to be the same, that isn’t business-crucial-same. So, what’s the alternative? Cut and paste, but that’s a lot of work and we often mess it up. 
This is my current golden-yak project: libbits. Maybe this will chew a corner off the nasty problem of dependency management. Atomist has automation infrastructure for modifying code, so I can make a program to do copy, paste, modify for a bit of code that’s handy in multiple projects. And then I can make a program to create that program. I’m following the yak stack, looking for the pot of gold at the end.

Then I’d like to make it easier to make the program that writes a program. Another thing we are making at Atomist is microgrammars, a new way of parsing, somewhere between Regex and a BNF grammar. (I don’t think it counts as a yak when your CEO tells you he wants your help with it.) My personal yak is to put an API on this that is super easy to use, and who knows, we might create a replacement for `sed`.

When are we justified in shaving a Golden Yak? If you don’t work for a development automation company, I don’t know. Getting an automation smooth enough that people use it like breathing is a lot of localized work for a huge but spread out benefit. Still, coding is fun! These yaks make great play.


There’s rarely a direct work-related reason to shave them well enough to find their deep secrets, the surprising wisdom on their shining skin under all that hair. The new way of working that is revealed only after we try it. Change our environment, and thereby change ourselves.

When I sit down to work on a weekend (or some Friday afternoons), I let myself dig into some Trim Yaks, and see how far I can take them. When I do this, I’m studying my own work. As automators, we study our users’ work, right? how they do their jobs. When we automate our own work we learn about our work deeply. Writing code to change code teaches me about code.

Here’s a shortcut to Golden Yak wisdom: cultivate an awareness of what we’re doing while we do it. I learned this trick from a book about how to ride a motorcycle: Devote 5% of your attention to how the work is going, and where it could be smoother. This awareness, then, feeds into our prioritization. And into understanding of our teammates’ experiences.

Conclusion

Yaks are a part of life and of progress.

Pair: We aren’t in this alone, and we can do it better together. Pairing lets us keep perspective on what we’re doing and why. We can use that space to think about doing it better.

Proven usefulness: look for it. Don’t spend a ton of time until you’ve hit a problem a ton of times. Track your yaks and timebox your attacks. Remember it’s about generativity, not all about you. Fill potholes, and shave a little more so your team can share the benefit.

Play: Stay curious. Especially learn about the system you work in, both socially and technically. Learn a little extra each time, about your tools, your system, and your companions on this journey. Interact and automate for the wisdom; for changing us, and thereby the world.

Yaks. They’re so much more than hair.

The Royal Yak

(continued from Trim Yaks; part of the Taxonomy of Yak Shaving series)

Royal Yak, aka Yakkity Yak (quote from yakbreeder.com)

Talking to people is yak shaving; it is an intermediate task that helps you get your official tasks done. It’s usually seen as a separate way to “waste” time. I’m here to call relationship-building out as a productivity booster.
Conway’s Law says that our system architecture resembles our communication flow (approximated by our org chart). Indeed, the human relationship graph has a close relationship to our system relationship graph — and if it doesn’t, that’s going to be a dysfunctional hunk of software.

Those interludes of chit-chat while you’re pairing? They matter. Especially on a remote team, when we don’t have chit-chat in a kitchen. Here are three crucial objectives for talking to people at work:

The more we know about what people know, the more we know whom to ask for help. This is the first thing I ask about when I meet a new person in the company. If my service interacts with another one, it helps to know who understands that one and can expound on its history. If I need to change nginx config, it helps to know that person who likes it.

Asking for help is a beautiful thing: it gets you help, and it gets you friends. Counterintuitively, people like you more if they have helped you, if they feel like you owe them one. This is great, especially with people outside the team, whom you can’t all maintain a close relationship with.

We need to know what our teammates don’t know. In particular, on a close team — and this is why a close team is limited to 7 people — we need to track what each other person knows and doesn’t, so that we can tell them the pieces of information they need. We have our own mental model of the system, our model of each other, and our model of each other’s models — all this is necessary for close coordination. This is how we avoid the potholes of communication failure. This is how we get distributed decisionmaking with trust: because we can predict each others’ actions. For smooth work, we need to know each other this well.

Let our teammates know that we want to know what they know, we want to hear their input. Psychological safety. We each need to feel safe suggesting ideas and asking questions. Google found that this is the #1 determiner of team effectiveness. So part of generativity is making your team a safe place for everyone to contribute. Do this by asking people for input and listening carefully to what they say. We don’t to agree; we have to welcome the information.

Social yaks are yaks. Get to know people, find out what they know, and make them feel safe. Keep them up to date on what you know.

One way I do this is with my own slack channel, #jessitron-stream. I use it as a notebook. I throw error messages in them when I hit them, and record what I did (especially when changing infrastructure). I track my own yak stack. People can pop in and answer questions if they want, or I can @ them and then they see the context of my inquiry. Lots of other Atomists have started their own stream channel; we have #radio-russ and #kipz-corner and #sylvain-stream and #hoffhaus and #arrrrr-day. It’s one way to work in public from the privacy of our own homes.

Don’t neglect the Royal Yaks. Hold court in Slack or in the kitchen. Interact with each other. Find out what is useful and what is in the way of other people, and you can prioritize your own yak shaving. These conversations, when they get interesting, might even lead you to the Golden Yak (ooooo).

Trim Yaks

(continued from Imperial Yaks; part of A Taxonomy of Yak Shaving series)

Trim Yaks, aka, the Hackhacking yak (making coding faster)

Each of the previous yaks stood in the way of a particular task. The Trim Yak is not so task-specific; these are the ones that let us work faster generally.

I nickname them “Hackhacking Yaks” after Lifehacking, which includes all the ways we change our environment in order to influence our own behavior. Change your world in order to change yourself, the feedback loop of human progress. Here, we’re talking about writing or configuring programs in order to change the way we code: Hackhacking. (It’s fun to say.)

At one extreme: how long between receiving your work laptop and when you can get work done smoothly? At least a couple days. There are essential Hackhacking yaks to shave here. Gotta install all the tools, gotta set up all the credentials. But then there’s more. Dotfiles, IDE configuration. For me, I cannot communicate in Slack until I’ve imported a few dozen essential emoji. To do that, I install a Chrome plugin and log in to Dropbox — Trim Yaks can stack up too.

These yaks may look like procrastination. And they can be, let run loose; configuring a keyboard shortcut can turn into writing a vim plugin.
For me, it’s stuff like, I have a library that’s used by another library that’s used by a service I want to test, which is a lot of installing and version-updating locally and doing that manually is just wrong and I want to automate it all! Or, something is wrong with my git repo, it’s in some weird state and I want to know how it got there and can I go read man pages until I understand it. Or, I hate changing nginx configuration because I don’t understand it, can I spend a day learning nginx configuration?

All of these things are useful. And they’re the kind of thing that when I do them, they pay off in ways I never would have expected. Automating the repository linkages (which I did on a weekend, for play) left me with a little toy program that later when I picked up Neo4J it was fast to implement a piece that makes a little dependency graph. Early in my git usage (this was 2012) I spent 6 hours (at work) watching a few videos and reading the book until I grasped the concepts. Later spent 45 minutes writing a talk that launched my speaking career. I still haven’t learned nginx configuration but gah! I so wish I had.
Trim yaks are about increasing our speed over time. It’s hard to know how useful a given Trim Yak will be; we might never use it again or we might use it a million times. Benefit is uncertain, so limit the cost.

Here’s a strategy:

Timebox! The first time something bugs you (“gosh, this code change could be faster, I bet the IDE has tools for this”), do it by hand. The second time, think about changing it: can you give it 15 min? that might be enough. If it isn’t, then the third or fourth time, give it an hour. Spend time only on what has proven useful.

And more considerations:

Stay curious! Learn a little bit more than you need to. I need one nginx option, I’ll look past the first StackOverflow answer, check the docs, and get a clue about why it works, or learn some new vocabulary word.

Pairing: I find pairing a big help with the Imperial Yaks; it helps to have someone else who is like “yeah we really have to fix this.” But it’s hard to shave Trim Yaks while pairing. Except! Pairing is a great time to share trimming tips. “I wish I had a faster way to do this, do you know one?” When your partner can help you get that IDE plugin set up, it’s efficient and also builds relationships.

Here’s the one Trim Yak that will pay off more than others: learning the system you work on. This is the one trick that separates hyperproductive developers from struggling ones: intimate knowledge of the system we work on. And it’s very, very hard to achieve in a system we didn’t write ourselves. Build your mental model at every opportunity. This yak that pays off most and fastest.

When pairing (or mobbing), learn what your partner (or everyone) knows.

When coding alone, dig around. Write tests, or break tests, or refactor. If you refactor, be careful: you built up your own mental model, your brain’s entwinement with the code, but you’ve invalidated the mental model other people have. Consider throwing the refactor away. Sometimes the team is like “this is a mess please change it,” but other times you’re not so much shaving a yak as combing the hair in a different direction.

Extract more value: Trim yaks increase personal productivity. If we take them a little farther, they can be generative. Post useful scripts or shortcuts in Slack. Document how to develop and test. If you do enjoy nginx configuration, tell people that you’re interested in it. They’ll be happy to share their nginx yaks with you next time, and you can build that skill. Blog! Those little tricks that you don’t use very often, let Google find them for you next time, and save hundreds of developers from your pain.

Share what you learn, and find out what’s holding other people back. You can do that with the next yak… the Royal Yak!

The Imperial Yaks

(continued from Attack Yak; series begins with Taxonomy of Yak Shaving) Sometimes you’re coding along, writing tests as little experiments “this should fail because I haven’t implemented the parser for it yet” — and it fails in a way you didn’t expect. And then you start digging and the parsing library isn’t working how you expected. And that library (it’s our library) doesn’t have a test for this case. Now you need to switch your development environment over. The yak stack is getting deeper. You have encountered …

The Imperial Yaks!

Imperial Yaks, aka the Yak Stack; quote from yakbreeder.com

These are the yaks that stack on top of each other. They all deserve to be shaved. This is a real bug in the library. It needs more tests. Then fix the bug, then deal with the linter and make a PR and wait for its review and do a release… and in the meantime to move forward we must figure out how to get npm to use local dependencies, which is not trivial with TypeScript.

How far do you go? When do you look for a workaround — maybe we can parse this with regular expressions and not use our library. When do you ask “Is this change even worth making?”

Pairing helps. I love pairing because I can be down in the weeds of the syntax and the particular bug while my pair still has part of her brain operating at the level of “why are we doing this again?”

If your original task is urgent: look for workarounds sooner. Never abandon the yak silently; create an issue if you don’t have time to fix it. Leave enough information to reproduce the bug. (This is work.) At the very least, throw something into Slack for tracking.

If the yaks you uncovered are dangerous: they might be more important that your original task. Realize you’re doing something important. Update your team and re-orient.

In the example about that failing test, it’s Rod who hit that last Saturday. He didn’t get his particular goal accomplished over the weekend. But he fixed a bug that was going to hold Clay back. He added tests. I popped in and paired on it, and added a few more pending tests for corner cases, which Rod later fixed. Sylvain merged the changes and did a release. All of us are more concerned with Atomist as a whole than whatever assigned tasks are on our list. That’s our culture.

When I aim for generativity, I can think more broadly about yaks. I can decide whether the yak stack confronting me is worth defeating, or going around. I feel good about creating an issue, if that’s all I can do right now: a warning sign to other people. Or leaving the code a little clearer than I found it: more null checks, an extra test, type declarations (in gradually-typed languages like TypeScript or Clojure).

My yak stack is now deeper than it used to be. Every confusing error I get, I solving it until I’ve made the error more communicative, seen the new message, and then committed that. Or changed the README. When I started at Stripe, after 4 months I felt like I wasn’t getting enough work done, and when I expressed this to others — because when I’m freaking out, it’s my job to ask for help — Julia and Danielle were like, but you’ve been making so many commits! Some managers at some companies look only at assigned tasks, but you can bet all my coworkers saw those README and error message and infrastructure changes and thought “Thank you!” about each one. Stripe has a culture of generativity, so I was fine.

It isn’t just about me and my personal productivity. Teamwork isn’t a race, with each person trying to be faster than the rest of the team.

Yak Racing

In the past, I’ve worked with developers who hoarded information, excluding others from customer meetings, insisting on a code structure that makes sense to them and no one else, turning work into a competition with the rest of us.

This exclusively-personal-maximization is also a danger of the next kind of yak: the Trim Yak.

The Attack Yak

(This post describes the first yak category. You might choose to start with the Taxonomy of Yak Shaving intro.)

Black Yak, aka Attack Yak. (quotes are from yakbreeder.com)

When you feel that your task is 80% complete, these yaks form the next 80%. They are the distance between “works on my box” and “this is done.”
All the data setup. Verification. Approvals. Status checks. Linters and code review. These are the yaks we have summoned voluntarily. There are also surprises like, I’m ready to integration test, but the dev environment is broken. Or merge conflicts. Or an unrelated test failure. Or someone wants to argue about whether this change is even a good idea.

Attack yaks are very frustrating, but there’s no getting around them (without cheating). You have to shave them. They make our development flow rough, by forcing us to context switch. What can we do?

Be cautious about summoning these yaks. We choose to all write automated tests. It doesn’t make us faster today, but it makes our development smoother in the medium and long term. Except when it doesn’t; intermittent failures, especially in tests that are too wide to be diagnostic, can make development rough instead. Many fine-grained tests prevent refactoring, or enforce functionality that isn’t needed anymore but nobody knows it. Be conscious in your yak shaving. Don’t shave its eyebrows.

On some projects, we summon the yak of code reviews. What is the objective? If it’s code quality, pairing is live, yakless code review with no context switches yeah! If it’s keeping up the team’s mental model, then keep reviews focused on that; discourage nitpicks. As a reviewer, if you want to see another test, add it. Give me a commit instead of an attack yak.

:attack-yak:

Track the attacks. We have a yak emoji in Slack for these.

:attack-yak: Staging environment is down, I can't test

If we all do that, we can search Slack before the retrospective to find out which ones are interrupting us the most. Our retro can have a little more data than our gut feel, which is really based on the last few days.

Coordinate your attack. Once we know which yaks are holding us back, shave them with automation. Shave them until they’re not an interruption anymore. Autofix on the linter. Script creation of the dev environment, so that “it’s broken” means “recreate it” instead of “figure out what the problem is today and create a special custom fix.” Then automate regular re-creation.

This work can feel like a waste of time, because it isn’t getting features delivered. Yet, if we think about our work as more than personal productivity, we see the value. On my team, we care about generativity (as opposed to productivity): the difference in what your team does with you, vs what it would get done without you. It’s about the team, not about me. And it’s about the team in the future, not only right now. Not about going faster, but about going smoother: removing obstacles for everyone, present and future us.

Every time you hit a build or environment failure, ask yourself, “How could this be easier to diagnose?” It might be “put a better error message here.” Fill the potholes you encounter. Smooth the path for future-you and future-team.

Sometimes you’re coding along, writing tests as little experiments “this should fail because I haven’t implemented the parser for it yet” — and it fails in a way you didn’t expect. And then you start digging and the parsing library isn’t working how you expected. And that library (it’s our library) doesn’t have a test for this case. Now you need to switch your development environment over to work on that (have you automated that?). The yak stack is getting deeper. You have encountered …

The Imperial Yaks!

A Taxonomy of Yak Shaving

Yak Shaving is: doing seemingly-unrelated tasks to get some real task done.

The name has obscure roots and doesn’t matter; the key is that it makes you think, “Why would you do that?”

Yak shaving can be: I want to make dinner, but I’m doing stairs; moving objects around in order to clear a path to put away the blender which is on the counter in the spot I need to unload the groceries. And then I’ll search through my email to find the name of the recipe so I can google it, and there in my email there will be something urgent to respond to, it’ll make me check my calendar which will remind me that I have to be somewhere tomorrow morning which means I need to tell my husband right now that I’m going to need the car, which starts a discussion. The milk is getting warm on the counter. These are all yaks: they’re not what I want to accomplish, but they are on the way to cooking dinner.

Oh and then the recipe says, “Beat the egg whites until stiff” and I look up what that means. I learn that the best tools for beating egg whites are a copper bowl and a balloon whisk — do I go buy these now? Gathering and learning the proper tools is a yak shave, too.

In programming, yak shaving has a reputation as hoop-jumping or time-wasting. Sometimes it is, when I’m shopping for the perfect keyboard under shadow of a promised delivery date. Other yak shaves are key to a smooth development process: activities that save you production outages, that spare your team days of troubleshooting, that guide your users toward the happy path and make them love your app.

There’s a balance here, between making the task doable and forcing the task to be done. Between scrubbing the counters and unloading groceries on the floor. Between researching top balloon whisks, and using a plastic whisk that’s dirty from last week.

Let’s categorize these yaks. Then we can talk about when to fight them, when to avoid them, and when to lovingly shave them clean. Because some of them are worthy opponents! Some of them are distractions, and some of them have secret wisdom on their skin underneath all that hair.


Yakbreeder.com lists five varieties of domestic yak. I’ll use them to categorize the yak shaves that we might do while programming, so that we can decide which ones to skip, and how to tackle the others.

The goal is a team development flow that moves forward while getting smoother every day, more predictable and more fun.

There’s a lot to cover, so this is a series. Start with: the Attack Yak!