Knowledge resides in teams

The magic of a gelled team is that they know how to work together, and together, they know how to do particular work. The members don’t know how to work together; the team does.

These learnings don’t reside in the members individually, the learnings are in the interrelations.

A shared know-how is jointly constructed between the participants. This shared know-how does not amount to the sum of the individuals’ know-hows nor does it strictly “belong” to any of the participants…. It involves instead the practice of coordinating sensorimotor schemes together, navigating breakdowns, and it belongs to the system the participants bring forth together: the dyad, the group, the family, the community, and so on.

“Linguistic Bodies The Continuity between Life and Language” Ezequiel A. Di
Paolo, Elena Clare Cuffari, and Hanne De Jaegher, quoted by @theblub

If you’re a director who gets to decide which teams stay together and which break apart, you have a lot of power — and very little control. Power can bust up symmathesies, but not build them or repair them. Other levels of hierarchy can set up conditions for success, but teams grow from within.

Leaving a company is scary because we know how to be in that company. Our own knowing-how-to-be exists partly in our interrelations there. Finding a new job means discovering a new way, a new self, to be in the new place. With families, even more so – this is part of what makes divorce so scary. If you’re in an unhealthy system and can’t imagine anything else, this is normal.

When you do get to be part of a healthy team, or a healthy family, appreciate it. Cherish it and nourish it.

Teams are like bread

Maybe when companies make you do “team-building” activities, what they’re looking for is a phase transition into a gelled team. Because it is a sudden, magical thing, right? When a group of people turns into a team.

Once you get there, to that feeling of team, it’s self-reinforcing. You trust each other, so y0u don’t take miscommunications personally; you work to restore communication, and so trust increases. You understand each other, so it’s easy to build further understanding. Working together gets smoother and smoother.

But how do you get there? pfft. How do you make sourdough bread? You grow it from sourdough starter, which you got from … someone else’s starter. Or from putting sugar water out on a windowsill and hoping some yeast lands in it. Seriously.

Will Larson suggests that when you have a gelled team, keep it. If you need to adjust how many people are helping with which pieces of software, then shift responsibilities from one team to another, not people.

When you have a gelled team, you can grow it gradually. Let the team reform with the new member incorporated before adding another.

Will suggests that when you want another team, gradually grow a gelled team up to 8-10 (max size) and then fork it into two teams of 4-5 (minimum size). It’s kind of like the sourdough starter: grow it, divide it, make the bread. Keep it alive the whole time.

If you have one team where the magic is flourishing, don’t kill it. Feed it, grow it, and let it be a source of further strong teams. No rushing.

Otherwise – if you take the group to paintball, or get them to mob program, or put them in a team room with sugar and water, maybe the yeast will blow in?

(if you want these little posts as I make them, plus a bit of extra context, sign up for my newsletter!)

Respond to input? No, we create it

Naturally intelligent systems do not passively await sensory stimulation.


They are constantly active, trying to predict (and actively elicit) the streams of sensory information before they arrive…. Systems like that are already (pretty much constantly) poised to act.


We act for the the evolving streams of sensory information that keep us viable and serve our ends… perception and action in a circular causal flow.

Andy Clark, Surfing Uncertainty

This book is about how human brains perceive based on predictions and prediction errors. I used to think we take in light from our eyes, assemble all that into objects that we perceive, and then respond to those perceptions. But it’s different: we are always guessing at what we expect to perceive, breaking that down into objects and then into expected light levels, and then processing only differences from us that expectation. We turn that into possibilities for our next action.

We don’t sit around waiting to notice something and then respond! We’re always predicting, and then thinking what would we like to see or explore next, and then acting toward that imagined reality.

Teams are intelligent systems like this too. Development teams (symmathesies) operating software in production have expectations of what it should be outputting. We are constantly developing better visibility, and stronger expectations. Then we want that output (data, usage volume) to change, so we take action on the software.

Teams and developers, as intelligent systems in conjunction with software, need to choose our own actions. Acting and perceiving are interlinked. Some of that perception is from the higher levels of organization, like: what does our company want to accomplish? What are people asking us to change? But in the end we have to act in ways we choose, and look for differences in what we perceive, to learn and grow.

Notice that “act as I choose” is very different from “do what I want” or worse, “do what I feel like doing” (which rarely corresponds to what will make me happy). I choose what to do based on input from people and systems around me, according to what might be useful for the team and organization and world.

If my boss wants something done in the code, they’d better convince me that it’s worth doing. Because only if I understand the real “why” can I predict what success looks like, and then I can make the million big and tiny decisions that are the work of software development. Does memory use matter? What is permanent API and what internal? Which off-happy-path cases are crucial, and how can I make the rest fall back safely? Where should this new function or service live?

If I choose to do the work only because they said to, only so that I can check off a box, I am not gonna make these umpteen decisions in ways that serve future-us.

Our job is making choices. We need to the background and understanding to choose our high-level work, so that we can make skillful choices at the low levels.

Intelligent systems don’t wait around to be told what to do. We are constantly looking for the next input that we like better, and creating that input. Act in order to perceive in order to act. This is living.

Reuse

Developers have a love-hate relationship with code re-use. As in, we used to love it. We love our code and we want it to run everywhere and help everyone. We want to get faster with time by harnessing the work of our former selves.
And yet, we come to hate it. Reuse means dependencies. It means couplings. It means surprises, when changing code impacts something we did not expect, or else it means don’t touch it, it’s too scary. It means trusting code we don’t understand because it’s code didn’t write.

//platform.twitter.com/widgets.js
Here’s the thing: sharing code is dangerous. Do it sparingly.

When reuse is bad

Let’s talk about sharing code. Take a business, developing software for its employees or its customers. Let’s talk about code within an organization that is referenced in more than one service, or by multiple flows in a monolith. (Monolith is defined as “one deployable unit maintained by more than one small team.”)

Let’s see some pictures. Purple Service here has some classes or functions that it finds useful, and the team thinks these would be useful elsewhere. Purple team breaks this code out into a library, the peachy circle.

purple circle, peach circle inside

Then someone from Purple team joins Blue team, and uses that library in Blue Service. You think it looks like this:

peach circle under blue and purple circles

Nah, it’s really more like this:

purple circle with peach circle inside. Blue circle has a line to peach circle

This is called coupling. When Purple team changes their library, Blue team is affected. (If it’s a monolith, their code changed underneath them. I hope they have good tests.)
Now, you could say, Blue team doesn’t have to update their version. The level of reuse is the release, we broke out the library, so this is fine.

picture of purple with orange circle, blue with peach circle.

At that point you’ve basically forked, the code isn’t shared anymore. When Blue team needs to make their own changes, they first must upgrade, so they get surprised some unpredictable time later. (This happened to us at Outpace all the time with our shared “util” libraries and it was the worst. So painful. Those “timesavers” cost us a lot of time and frustration.)

This shared code is a coupling between two services that otherwise have nothing to do with each other. The whole point of microservices was to decouple! To make it so our changes impact only code that our team operates! dead. and for what?

To answer that, consider the nature of the shared code. Why is it shared?
Perhaps it is unrelated to the business: it is general utilities that would otherwise be duplicated, but we’re being DRY and avoiding the extra work of writing and testing and debugging them a second time. In this case, I propose: cut and paste. Or fork. Or best of all, try a more formalized reuse-without-sharing procedure [link to my next post].

What if this is business-related code? What if we had good reason to DRY it out, because it would be wrong for this code to be different in Purple Service and Blue Service? Well sorry, it’s gonna be different. Purple and Blue do not have the same deployment schedules, that’s the point of decoupling into services. In this case, either you’ve made yourself a distributed monolith (requiring coordinated deployments), or you’re ignoring reality. If the business requires exactly one version of this code, then make it its own service.

picture with yellow, purple, and blue circles separate, dotty lines from yellow to purple and to blue.

Now you’re not sharing code anymore. You’re sharing a service. Changes to Peachy can impact Purple and Blue at the same time, because that’s inherent in this must-be-consistent business logic.

It’s easier with a monolith; that shared code stays consistent in production, because there is only one deployment. Any surprises happen immediately, hopefully in testing. In a monolith, if Peachy is utility classes or functions, and Purple (or Blue) team wants to change them, the safest strategy is: make a copy, use the copy, and change that copy. Over time, this results in less shared code.

This crucial observation is #2 in Modern Software Over-engineering Mistakes by RMX.

“Shared logic and abstractions tend to stabilise over time in natural systems. They either stay flat or relatively go down as functionality gets broader.”

Business software is an expanding problem. It will always grow, and not with more of the same: it will grow in ways you didn’t plan for. This kind of code must optimize for change. Reuse is the enemy of change. (I’m talking about reuse of internal code.)

Back in the beginning, Blue team reused the peach library and saved time writing code. But writing code isn’t the expensive part, compared to changing code. We don’t add features faster as our systems get larger and we have more code hypothetically available for re-use. We add features more slowly, because every change has more impacts and is less safe. Shared code makes change less safe. The only code safe to share is code that doesn’t change. Which means no versioning. Heck, you might as well have cut and pasted it.

When reuse is good

We didn’t advance as an industry by rewriting, or cut and pasting, everything we need over and over. We build on libraries published by developers and companies all over the globe. They release them, we reuse them. Yes, we get into dependency hell, but it beats writing your own web framework. We get reuse not only of the code, but of understanding: Rails knowledge transfers between employers.

There is a tipping point where reuse is magical.

I argue that this point is well past a release, past a separate jar.
It is past a stable API
past a coherent abstraction
past automated tests
past solid documentation…

All these might be achieved within the organization if responsibility for the shared utilities lives in a separate team; you can try to use Conway’s Law to enforce architectural boundaries, but within an org, those boundaries are soft. And this code isn’t your business, and you don’t have incentives to spend the time on these. Why have backwards compatibility when you can perform human coordination instead? It isn’t worth it. In my past organizations, shared code has instead been the responsibility of no one. What starts out as “leverage” becomes baggage, as all the Ruby code is tied to an old version of Sinatra. Some switch to Go to get a clean slate.
Break those chains! Copy the pieces you need out of that internal library and make them yours.

At the level of winning reuse, that code has its own marketing department
its own sales team
its own office manager
its own stock price.

The level of reuse is the company.

(Pay for software.)

When the responsible organization succeeds by making its code stable and backwards-compatible and easy to work with and well-documented and extensively tested, that is code I want to reuse!

In addition to SaaS companies and vendors, there are organizations built around open-source software. This is why we look for packages and frameworks with a broad community around them. Or better, a foundation for keeping shared projects healthy. (Contribute to them.)

Conclusion

Reuse is dangerous because it introduces coupling. Share business code only when that coupling is inherent to the business domain. Share library and utility code only when it is maintained by an organization dedicated to publishing that code. (Same with services. If you can pay for infrastructure-level tools, you’ll get better tools without distracting your organization.)

Why did we want to reuse internal code anyway?
For speed, but speed of change is more important.
For consistency, but that means coupling. Don’t hold your teams back with it.
For propagation of bug fixes, which I’ve not seen happen.

All three of these can be automated [LINK to my next post] without dependencies.

Next time you consider making your code reusable, ask “who will I sell this to?”
Next time someone (including you) suggests you reuse their code, ask “who publishes that?” and if they say “me,” copy it instead.

Tradeoffs in Coordination Among Teams

The other day in Budapest, Jez Humble and I wondered, what is the CAP theorem for teams? In distributed database systems, the CAP theorem says: choose two of Consistency, Availability, and Partitioning — and you must choose Partitioning.
Consider a system for building software together. Unless the software is built by exactly one person, we have to choose Partitioning. We can’t meld minds, and talking is slow.
In databases we choose between Consistency (the data is the same everywhere) and Availability (we can always get the data). As teams grow, we choose between Consensus (doing things for the same reasons in the same way) and Actually-getting-things-done.
Or, letting go of the CAP acronym: we balance Moving Together against Moving Forward.

Moving Together

A group of 1 is the trivial case. Decision-making is the same as consensus. All work is forward work, but output is very limited, and when one person is sick everything stops.
A group of 2-7 is ideal: the communication comes with interplay of ideas, and whole new outputs of dialogue make up for the time cost of talking to each other. It is still feasible for everyone in the group to have a mental model of each other person, to know what that person needs to know. Consensus is easy to reach when every stakeholder is friends with every other stakeholder.
Beyond one team, the tradeoffs begin. Take one team of 2-7 people working closely together. Represent their potential output with this tall, hollow arrow pointing up.
This team is building software to run an antique store. Look at them go, full forward motion. (picture: tall, filled arrow.)
Next we add more to the web site while continuing development on the register point-of-sale tools. We break into two teams. We’re still working with the same database of items, and building the same brand, so we coordinate closely. We leverage each others’ tools. More people means more coordination overhead, but we all like each other, so it’s not much burden. We are a community, after all.
A green arrow and a red arrow, each connected by many lines of communication, are filled about halfway up with work.
Now the store is doing well. The web site attracts more retail business, the neighboring antique stores want to advertise their items on our site, everything is succeeding and we add more people. A team for partnerships, which means we need externally-facing reports, which means we need a data pipeline.
A purple arrow and a blue arrow join the red and green ones. Lines crisscross between them, a snarly web. The arrows are filled only a little because of these coordination costs. The purple arrow is less connected, and a bit more full, but it’s pointed thirty degrees to the left.
The same level of consensus and coordination isn’t practical anymore. Coordination costs weigh heavily. New people coming in don’t get to build a mental model of everyone who already works there. They don’t know what other people know, or which other people need to know something. If the partnerships team touches the database, it might break point of sale or the web site, so they are hamstrung. Everyone needs to check everything, so the slowest-to-release team sets the pace. The purple team here is spending less time on coordination, so the data pipeline is getting built, but without any ties to the green team, it’s going in a direction that won’t work for point of sale.
This mess scales up in the direction of mess. How do we scale forward progress instead?

Moving Forward

The other extreme is decoupling. Boundaries. A very clear API between the data pipeline, point of sale, and web. Separate databases, duplicating data when necessary. This is a different kind of overhead: more technical, less personal. Break the back-end coupling at the database; break the front-end (API) coupling with backwards compatibility. Teams operate on their own schedules, releases are not coordinated. This is represented by wider arrows, because backwards compatibility and graceful degradation are expensive. 
Four arrows, each wide and short. A few lines connect them. They’re filled, but the work went to width (solidness) rather than height (forward progress).
These teams are getting about as far as the communication-burdened teams. The difference is: this does scale out. We can add more teams before coordination becomes a limitation again.
Amazon is an extreme example of this: backwards compatible all the things. Each team Moving Forward in full armor. Everything fully separate, so no team can predict what other teams depend on. This made the AWS products possible. However, this is a ton of technical overhead, and maybe also not the kindest culture to work in.
Google takes another extreme. Their monorepo allows more coupling between teams. Libraries are shared. They make up for this with extreme tooling. Tests, refactoring tools, custom version control and build systems — even whole programming languages. Thousands of engineers work on infrastructure at Google, so that they can Move Together using technical overhead.

Balance

For the rest of us, in companies with 7-1000 engineers, we can’t afford one extreme or the other. We have to ask: where is consensus important? and where is consensus holding us back?
Consensus is crucial in objectives and direction. We are building the same business. The business results we are aiming for had better be the same. We all need to agree on “Which way is up?”
Consensus is crippling at the back end. When we require any coordination of releases. When I can’t upgrade a library without impacting other teams in way I can’t predict. When my database change could break a system more production-critical than mine. This is when we are paralyzed. Don’t make teams share databases or libraries.
What about leveraging shared tools and expertise? if every team runs its own database, those arrows get really wide really fast, unless they skimp on monitoring and redundancy — so they will skimp and the system will be fragile. We don’t want to reinvent everything in every team.
The answer is to have a few wide arrows. Shared tools are great when they’re maintained as internal services, by teams with internal customers. Make the data pipeline serve the partnership and reporting teams. Make a database team to supply well-supported database instances to the other teams. (They’re still separate databases, but now we have shared tools to work with them, and hey look, a data pipeline for syncing between them.)
The green, red, and blue arrows are narrow and tall, and mostly full of work, with some lines connecting them. The purple arrow and a new black arrow are wide and short and full of work. The wide arrows (internal services) are connected to the tall arrows (product teams) through their tips.
Re-use helps only when there is a solid API, when there is no coupling of schedules, and when the providing team focuses on customer service.

Conclusions

Avoid shared code libraries, unless you’re Google and have perfect test coverage everywhere, or you’re Amazon and have whole teams supporting those libraries with backwards compatibility.
Avoid shared database instances, but build internal teams around supporting common database tools.
Encourage shared ideas. Random communication among people across an organization has huge potential. Find out what other teams are doing, and that can refine your own direction and speed your development — as long as everything you hear is information, not obligation.
Reach consensus on what we want to achieve, why we are doing it, and how (at a high level) we might achieve it. Point in the same direction, move independently.
Every organization is a distributed system, even when we sit right next to each other. Coordination makes joint activity possible, but not free. Be conscious of the tradeoffs as your organization grows, as consensus becomes less useful and more expensive. Recognize that new people entering the organization experience higher coordination costs than you do. Let teams move forward in their own way, as long as we move together in a common direction. Distributed systems are hard, and we can do it.


Bonus material 

Here is a picture of Jez in Budapest.

And here is a paper about coordination costs:
Common Ground and Coordination in Joint Activity

Fitting in v. Belonging

In your team, do you feel like you fit in? Do you have a feeling of belonging?

These are very different questions.[2] When I fit in, it’s because everyone is sufficiently alike. We have inside jokes, TV shows or sports we talk about, opinions we share and common targets of ridicule. New people can fit in by adopting these opinions and following these sports.

When I belong, it’s because everyone wants me to be there, because the group wouldn’t be the same without me. We value each other for our differences. We have values that we share, and opinions we discuss. New people are integrated as we come to know and appreciate each other.

“Fitting in,” that superficial team culture of common interests and inside jokes, is much easier to establish. And it’s easier to hire for, because we can base choices on appearances and social cues. Hoodies and quotes from The Princess Bride. But it doesn’t get us a strong team. On a strong team, people share ideas, they pull from their varied perspectives, they emphasize their differences because that’s their unique contribution. This weaving together of various strengths, respect for the unexpected — this is how a team can be stronger than its parts, this is where novel solutions come from. It emerges from feelings of belonging, which come from the group’s deeper culture. That’s much harder to establish.

On a wholehearted team, we show up as our whole selves and put all our creativity into the team’s goals. How can we achieve this? Hire for value fit, not culture fit. Don’t settle for the comfort of “fitting in” – aim for the safety of belonging.

I had this kind of team at my last job, at Outpace. We loved each other as people and respected each other as developers. And this was a remote team – we didn’t fall back on physical proximity as appearance of teamwork. We shared goals, discussed them and evolved them. We shared our frustrations, both work and personal. When opinions clashed, we asked why, and learned. On this team, I explored ideas like the sea map. We grew individually and together.

That feeling of belonging makes it safe to express ideas and to run with them. And to take ideas from others and expand on them. Poof: innovation. Without that feeling of belonging, when the aim is to fit in, we express agreement with dominant voices.[1] Superficial cultural fit actively represses new ideas.

How can we move our teams toward a greater sense of belonging? Ask people about their interests that you don’t share. Respect each person’s experiences and opinions, especially when these are unique among the group. Instead of “We all agree, so we must be right,” say, “We all agree. This is dangerous; can we find another view?” When pair programming, if you think your pair has the wrong idea, try it anyway. When someone says something dumb, their perspective differs; respond with curiosity, not judgement. Cherish our differences, not superficial similarities. Sacrifice the comfort of fitting in for the safety to be ourselves.


[1] Research has shown that teams of similar-looking people emphasize similarities. They’re driven toward groupthink, quiet silencing of dissent. When someone breaks the uniformity, the not-obviously-different people starts expressing the parts of them that are unique. (I can’t find the reference, anyone know it?)

[2] The dichotomy between fitting-in and belonging comes from Brené Brown’s book, Daring Greatly.

Data v Awareness

In the computer industry, data and conscious thinking are praised, as opposed to an integrated awareness.[1] How is the work going? the task-tracking tools, the commits, and the build results provide data, but only conversations with the team can provide awareness. Awareness of mood and relationships and trends, of uncertainties and risks. Perhaps this is part of organizations’ fear of remote work: colocation provides opportunities to read the mood of the team. Data alone can’t provide that.

In sociological research like Brené Brown’s, she starts with awareness: interviews, a person’s story in context. Then she codes (in this context, “to code” is to categorize and label) the answers, and they become data. She aggregates that data to get a broader picture, and that leads to a broader awareness.

The key is: local awareness, to data, to aggregated data, to broader awareness.

On my last team, we were working on this. I wanted to track what was holding us back, and what was helping us move. Which tools in our technology stack cost us the most energy, and which improvements are paying off. To do this, we started posting in Slack whenever something frustrated us or helped us along, with custom emoticons as labels. For instance:

weight: clojure set operations behave unpredictably if passed a vector; lift: test-data generation utility for X service; weight: local elasticsearch version different from prod

This turns our awareness of the current situation into data, which a program can aggregate later. At retro time, I turned the words next to the hot-air balloon (“lift,” because it helps us move the project up and forward) into a word cloud.[2] The words next to the kettlebell (“weight,” because it’s weighing down the balloon, holding us back) formed a separate word cloud. This gave us a visualization to trigger discussion.

The aggregation of the data produced a broader level of awareness in our retrospective. This contrasts with our remembered experience of the prior sprint. Our brains are lousy at aggregating these experiences; we remember the peak and the end. The most emotional moment, and the most recent feelings. The awareness -> data -> aggregation -> awareness translation gives us a less biased overview.

The honest recording of local awareness happens when the data is interpreted within the team, within the circle of trust, within context. There’s no incentive to game the system, except where that is appropriate and deliberate. For instance, the week after the first word cloud, Tanya posted in the channel:

weight: elasticsearch elasticsearch elasticsearch elasticsearch elasticsearch

She’s very deliberately inflating a word in the word cloud, corresponding to the level of pain she’s experiencing. (context: we were using Elasticsearch poorly, totally nothing wrong with the tech, it was us.) Her knowledge of how the data would be used allowed her to translate her local awareness into a useful representation.

Data alone is in conflict with a broad, compassionate awareness of the human+technological interactions in the team. But if the data starts with awareness, and is aggregated and interpreted with context, it can help us overcome other limitations and biases of our human brains. In this way, we can use both data and awareness, and perhaps gain wisdom.

—-
[1] “Computing: Yet Another Reality Construction,” by Rodney Burstall, inside Software Development and Reality Construction
[2] Thank you @hibikir1 for suggesting the first reasonable use of a word cloud in my experience