Automation story: GraphQL schema deployment

Building a new automation to give power to my team

At Atomist, we supply a GraphQL endpoint to reveal links between commits, builds, deploys, and people. Internally, we also use GraphQL to access data in Neo4J. To do that, we need to deploy a schema definition (IDL) to the Neo4j GraphQL extension.

Software evolves from simple to rigorous. In the experimental stage (“do we want to use this?”) I constructed a schema.idl and pushed it to Neo4J by hand. One person doing all the schema is okay for a little while, but not for production. I want other people to improve the schema, too! So I automated its path from a GitHub commit to the database. Now any member of our team can deploy a new schema, and they’re even prompted to do it in Slack after they push a change to schema.idl.

This automation lives in an Atomist automation client, which turns a program I could run locally into one my whole team can run. I pushed it up to the cloud, and now the whole team has my power to deploy the schema.

First, I made a command that retrieves the file from GitHub and posts it to Neo4J. The passwords and URLs are all in one place (my automation deployment), so people don’t need their own database login to upgrade the schema. The command can be triggered in Slack, or from a REST API (web form coming soon). Now everyone can deploy the schema.

I type “@atomist deploy graphql schema” and Atomist reports in the relevant channel that I did it.

This replaces two account creations, three logins, two curl commands, and some cryptic JSON interpretation with one “@atomist deploy graphql schema” message in Slack. Then I made it even better.

Next, I made an automation that recognizes when the schema.idl file changes, since that one is significant. It reports that significance back to Atomist, and this shows up in Slack as part of the standard Atomist report about commits pushed to GitHub. This is a way of noticing which commits affect different parts of the system.

Atomist reports a commit on the test-graphql-deploy branch; that the commit has a change to the graphql-schema, that two builds are in progress and there are no other blockers.

Finally, another automation responds to that change in schema.idl and pops a Slack message into the channel with a button! Push the button, and that “deploy graphql schema” command updates the schema in Neo4J, either the staging database (if you pushed the change to a branch) or production (if it’s mainline). Super convenient!

Atomist says, The GraphQL schema was updated in branch test-graphql-deploy. there’s a button “push to staging.” I pushed it. Atomist reports that I updated the schema.

This small piece of infrastructure matures our internal GraphQL implementation; it’s trackable, repeatable, safe, and quick. It’s easier for me, and a new power for the rest of the team — they didn’t even bother touching it before, and now they can. I love turning complicated READMEs into happy, timely buttons in Slack!

You can make a big piece of software more mature with a small automation.

Find the code for my automation here. If you’d like to hear more details about how to implement this example, so that you can do this kind of automation on your team, clap for this post or ping me in the Atomist community Slack. (Or get started right away with a Quick Start.) I’d love to hear from you.

Folding Tradeoff Space

automation changes the rules we play by

Fast v Safe

Tradeoffs are a fact of life. Choose between safety and speed, stability and flexibility. More of one sacrifices the other.

fast or safe, we are somewhere on that spectrum

I remember a time when this described our decisions about software deployments. Deployments caused problems, so we did fewer of them.

Then came Continuous Integration and Continuous Deployment. With a lot of work, we get automated tests and monitoring, automated deployment and rollback. Then we learned that small deployments are safer — so much safer that many fast deployments added up to less danger than one bulk deployment.

Fast and Safe

It’s like, this automation plus culture change has folded space — these arrows now point in the same direction.

fast and safe point in the same direction, they are no longer in conflict

Bending tradeoff space like this wasn’t free.

Tests double our application code, and infrastructure adds even more code. Harder: we had to change the culture. Think in smallest shippable units. Alleviate the next bottleneck after deploys aren’t it, and the next.

Without weeks of manual testing, we had to create new ways to answer “Does it work?” Without ops people standing by, we needed new ways to answer “is it up?” When each person makes deployment decisions individually, they need tools to support the decision with visibility, and tools to implement the decision with fast rollback in case of surprise.

This automation is expensive, but we wouldn’t go back to the old days of dangerous, slow deployments.

Flexible v Consistent

I want to talk about another tradeoff space, this one in “Do we create custom internal software?” As a business, I want internal software to increase the consistency, legibility, and speed of my employees’ work. We think of the cost of software in terms of money and time, work to create it and to keep it happy. There’s a deeper cost to the organization, though: ossification.

spectrum of flexible or consistent

The consistency enforced by software is great for managers and administrators; they get legibility, meaning they can track work at a high level. These are the people making decisions about software requirements.

I once worked on workflow tracking software for a research organization. The heads of the development effort bragged that they’d encoded the knowledge of the scientists, so they could leave and we’d still have it. The research could proceed… in exactly the same way. Any substantial change took (this is extreme, but it was ten years ago) eighteen months to roll out. So much for scientific innovation! oh but it was great for the higher-ups tracking progress of each experiment as it moved through this regimented system.

This consistency is not optimal for learning, not for daily patient care or customer service. People are adaptable, but software inhibits this adaptation. Ask any nurse, social worker, or hotel receptionist how they feel about the software they have to use. (Ask them when they are off duty.) They’ll tell you about frustrations, time wasted, obstacles to doing great work.

Compared to a paper form, an electronic form is constraining. Compared to talking to each other, coordinating via task trackers is limiting. Software ossifies processes. It impedes learning and improvisation.

Flexible and Consistent

How could we fold this tradeoff space?

This tradeoff exists when the people using the software can’t change the software. Maybe it can help to look at one place where this tradeoff does not exist, where it is naturally folded.

flexible and consistent in the same direction: local automation

When the users are also the system movers, calcification doesn’t happen.

Software teams developing for themselves gain consistency, legibility, speed and more — while adding flexibility. When deployment is automated in one place, we can improve the process without the work of disseminating information.

Automation helps with testing. It helps with deployment. What else can we automate for ourselves? feature tracking, dependency updates, information gathering, project creation, common code change tasks?

Automation isn’t cheap, but it is priceless. Let’s smooth our own work with comprehensive automation. We might even get better at moving systems and help the whole business flex.

Effective Use of Slack

A lot of us are using Slack now. Some of us are past the shiny and into misery, where we wish for the good old days of email and talking with our mouths.

Let’s look at this technology thoughtfully, and see whether we can corral it into helpfulness. Dan North gave a great talk about Eli Goldratt’s The Goal, which teaches how to take advantage of and prevent harm from a new technology. He included examples from manufacturing, accounting, and DevOps. Let’s use this technique for Slack.

Four questions to ask about any new technology:

  1. What is the power?
  2. What limitations does the technology diminish?
  3. What rules enabled us to manage this limitation? (gotta throw those away, or you get no benefit)
  4. What new rules will we need?

What is the power of Slack?

:power:

Slack brings instant, persistent communication to the masses. Compared to email, it’s fast. Compared to IRC, it’s accessible, bringing critical mass of network size. Compared to in-person communication, it’s persistent and searchable.

Slack also changes a decision point: with an email or a meeting invite, the transmitter of information chose the recipient. In public Slack channels, everyone can look at it, including people who weren’t even in the company when I posted it, hypothetically. This moves power from the writer to the reader.

What limitations does Slack diminish?

There are two varieties of communication to consider separately. One is informal discussion, which used to happen in person. This requires speed or response. The other is broadcast, which happened over email or in speeches. This is supposed to be received by everyone.

Now, with Slack, informal discussion can be fast and spontaneous without sound transmission between people in the same room. That’s a release of space limitations. Now, informal discussion doesn’t require you to break someone else’s concentration. You can ask a question, and they can look when they hit a stopping point. That’s a release of time limitations. Then there’s an even bigger release: you don’t have to be in that room at that time in order to overhear the conversation. The discussion and decision is available to the rest of the team whenever they get to work, in whatever time zone they work in.

:puke-rainbows:

Broadcast was once limited to speeches or emails — or before that, memos. These take time to compose. Slack messages are faster: throw them out there. You can get immediate feedback from everyone, potentially. Broadcasting to the whole company, to anyone who subscribes (now or later), is cheap and easy.

What rules enabled us to manage this limitation?

We had communication mechanisms before Slack, I swear we did.

For discussions we know we needed, we scheduled meetings and reserved conference rooms.

:coffeetime:

For discussions we didn’t know we needed, we had the kitchen, with its coffeemaker. Smoke breaks, back in the day. Shared space and time: physical colocation and standard work hours.

For broadcast, we had the hierarchy. Decisions and vision statements passed down from executive to manager to manager to manager to worker. Then results passed upward in the reverse hierarchy of status meetings: everyone sit here while each of you reports to me on your individual work.

Now that we’re free of these limitations, we can work anywhere, on any schedule, and we have transparency, so everyone can learn what’s happening from the CEO and CTO, and we can all know what the most important thing for us to work on is because we have all the information, right?

What new rules do we need?

Clearly we don’t have all the information, because Slack contains far more information than we can each absorb.

:gah:

What did we lose when we lost the old limitations? When throwing things out there became cheap and easy?

We lost curation. More information is not better — it isn’t even more. When there’s more out there than we can take in, we take in less. When every time a nurse brings up a patient record they get eight warnings, they stop reading any of them. When we log in to Slack and all the channels have dozens of unread messages, we hit Shift-Esc. At least I do, so that the channels all unbold and I can think again. There is value in transmitting less, in deciding what is important before you send it, so that everyone else in the company doesn’t have to decide for you.

:heart-and-mind:

We lost common ground. Common ground is shared knowledge that everyone knows that everyone knows. It’s a basis for efficient communication and trust in each others’ decisionmaking. When a team has its discussions in Slack, there’s an assumption that everyone heard everything. When someone was on vacation or troubleshooting production or just getting work done, they miss that, and no one else on the team knows they missed it. Or a manager writes something in a channel, they think everyone got it, and they’re wrong. We miss the feedback loop of seeing who is in the room, the grace of “oh right you weren’t there that day.” Thinking someone knows something they don’t leads to dangerous miscommunication.

We lost focused conversation. Asynchronous messaging is suited to multitasking. Multitasking is not suited to effective communication. Nor is the single dimension of ASCII text. Emojis help, but they can’t substitute for seeing each others’ faces. Nonverbal cues help transmit intention, and that prevents and heals hurt feelings. Also, focused dialogues and discussions help us develop shared mental models much more quickly. That means we need to talk synchronously: at the same time, in the same room (even if it’s a Zoom room).

At this extreme, more information is less.

What new rules do we need?

Right. So what conventions and guidelines can help us overcome this pain?

I haven’t got it all figured out, but I can give you a few that help at Atomist.

Verbosity is inversely related to Importance

Each channel has an expectation-of-reading level.

  1. There are a few “everyone reads all of this when they get back from vacation” channels, containing only brief updates about significant changes. There’s #announce for official announcements (including birthdays; it’s a small company). There’s #what-happened, where architectural or vision redirection is reported, usually including a link to detailed discussion in another channel.
  2. There’s one “everyone reads this daily but not when they come back from vacation channel” called #standup. These are the daily goings-on, what’s about to go to Prod, what objectives each person has, and what’s blocking them. We also have a 15-minute daily standup video call, which we record and post in #standup for people who are in other time zones or at the dentist that day.
  3. Then there are channels for items and questions that many people care about; #engineering gets “can someone help me debug this prod slowdown?” and “I’m about to restart the test database” and “I’m stumped and need a pair.” I check this channel when I’m online, but I won’t scroll back far when I get there in the morning.
  4. Each project has a channel for updates and chatter relating to it. We talk about and create issues, help each other troubleshoot, and answer questions like “Why did we do it this way again?” For updates on the code, @atomist bot condenses GitHub, Travis CI, and deployment notifications into tight, dynamically-updated messages. (You can get this too.) There’s no expectation of keeping up; each of us joins one or two of these channels while working actively on the project.
  5. Personal streams: I have a channel called #jessitron-stream that I use like a notebook. If I don’t have a pair, I’ll rubber-duck here. When I get a new error message, I’ll post it here and then track it down, and post the resolution. That’s handy for searching later. As a bonus, sometimes other people are poking around and jump in with “have you tried this?” When I need to ask someone a question, I’ll @-mention them in my channel, and then they can see the context of the question. Unless I specifically call for someone, there’s no expectation that anyone reads it. Google Calendar integration posts meetings at me, which is handy when someone wonders what I’m doing. Other personal streams in our Atomist slack include #tanya-coding, #kipz-corner, #radio-russ, and #aarrr-day.

Move to Zoom quickly

As soon as a discussion gets interesting, if it has all your attention or if it should, move it to Zoom (or your video conferencing choice, or in-person if that’s a thing). This improves concentration, increases information transfer, and removes a sneaky impression that everyone was part of the discussion when they weren’t.

:enhancement:

We have one Zoom room that’s a default for helping each other, asking questions, or generally hanging out while we’re working. We call it “bang-zoom” because our bot posts a link to it in response to “!zoom” in Slack. So when it’s time to share a screen or look each other in the eye, we type “!zoom” and head in there for face time.

Zoom for Safety

If something you read in Slack hurts, stop. Make a guess that the person didn’t intend it the way it came across, and get into a Zoom meeting ASAP to learn more detail. Your feelings are a useful clue that some information is missing on at least one side, so get that resolved and gain some common ground out of it.

:feelings:

We have a :zoom-for-safety: reactji. After I read something and then smack my desk in frustration, I use that reactji instead of responding on Slack.

Etiquette

Threads: if a channel’s conversation has moved on since the message we want to reply to, we’ll put that reply in a thread. No one is expected to read a thread they weren’t @-mentioned in.

Emoji: I need them. I love reacting without adding a message to the channel. When there’s something I want to say that doesn’t easily iconify: any image that comes up in a google search for that phrase is fair game. I’ll put some of my favorites at the end of this post.

Channel profligacy: so what if there are hundreds of channels? We join and leave, create and archive as our projects and people change. Crucial: just because someone is in a channel doesn’t mean they read it.

@-mention response time: This is asynchronous, people. If I want your attention, I’ll @-mention you; check Slack when you’re at a stopping point. I am not checking Slack while focused on writing this post. When I’m done, I’ll respond to all @-mentions and direct messages. If it’s an emergency, somebody has my cell phone number.

@channel: treated as an @-mention to everyone; read them.

@here: for something urgent right now (“I’m shutting down the dev environment for 15 minutes”) but not relevant later.

Private channels: useful for surprising someone for their birthday. Otherwise considered rude.

:tiptoe:

Direct messages: go for it, have a chat. When I want to ask @kipz a question about current work, I @-mention him in #jessitron-stream instead, in case anyone else wants to pipe in.

Conclusion

Slack is a technology like any other: useless unless you change how you work, painful unless you choose how you work.

At best, asynchronous messaging removes interruption; lets us search our history of solved problems; and catches us up on what we missed, giving freedom in time and space. At worst, it multiplies interruptions and hides information in a pile of cruft. It’s our choices that let us use it effectively, and stop it from using us.

Here are more of my favorite emoji.

:its-alive:

:oh-yeaahh:

:woo:

:y-u-no:

rofl

:irate-goose:

:yes-but-not-yet:

:scared-panda:

:party-parrot:

The famous party parrot has flashy colors that can trigger migraines or seizures. Be accessible! Replace it with this one.

:fart:

:rainbow-unicorn-butterfly-kitten:

Shaping our Tools to the Flow of Conversation: Issue Creation

Collaboration is hard. There are the essential difficulties of dealing with people, and then there are incidental challenges in the tools we use. Communicating with people, while clicking into to Travis to see the build status and asking: where is this deployed? Who tested it? I want to remove all those incidental challenges, even the tiny ones.

We collaborate with tools: Slack, an issue tracker, source control, many more. We want the best tool for each purpose, but switching between them is extra work for my brain. A tiny bit of extra work, but every distraction matters when I’m focused on a complex task.

For instance, we’re talking in Slack, and realize we need to track this with a GitHub issue. Sometimes two of us create an issue at the same time — oops! I want to create that issue right in the conversation where people can see it.

The Atomist bot can do this for me:

@atomist create issue

I don’t need all the functionality of GitHub Issues in Slack, just enough plus a link. This way I rarely have to switch contexts, and neither does anyone else there in the conversation. I can claim the issue and vote for it and add a label right here. If I pop over to GitHub and update the issue, the Slack message redraws when new information has come to light.

This is a small thing.

Small things matter when they let us keep more of our brain on the hard problems of coding and collaboration with people. Every tiny mental bump I can smooth out for my team makes each of us effectively smarter.

Create Issue is a tiny piece of Atomist’s built-in automation, interweaving the tools we have to suit the way we smoothly work. There’s more: commit messages are pretty and are integrated with build and deploy status, PR notifications are mergeable, even new projects are created in line with the conversation.

We don’t need a new interface for collaboration. We need to make the ones we have as perfect as possible, then customize per team.

As developers, we have the power to create the world we want: the world that gets the tiny problems out of our way so we can solve the hard stuff.

Spend some time automating your own work, with whatever tools you have. Invite the Atomist bot into your team to get Create Issue and other integrations. Join the development automation conversation in our community Slack. We’d love to hear how you shape your tools to fit your flow.

the sweet spot: Local Automation

When you learn to code, you acquire a superpower: automation. We turn the computer into a machine to do our (very specific) bidding. We get paid to automate what other people want. We can and should use this superpower for ourselves too!

Levels of automation

Are we programmers or software developers?

Probably both.

I think of a program as a piece of code that runs on my box and does something useful for me. It can be a one-line shell script or a Scala program backed by a graph database that downloads all my dependencies.

Software is also made of code. Lots of code, code on servers and code in browsers and code on mobile devices. Software runs on all the computers and is useful to millions of people. It has compatibility, integration, redundancy, persistence, security, tests, user experience, multi-region back ends and multi-platform front ends. It has program managers and technical writers and designers and operations. It has stakeholders.

Scaling and safety and compatibility make software super expensive: thousands or millions of times more expensive than a program. And then there’s the risk: will the Software be useful? (If it’s internal software people are required to use, it might even make their jobs harder. Net negative benefit.) What changes will make it more useful?

Optimization

Software and programs are degrees of automation. We build this automation in order to make a system work more smoothly. Optimizing that system is easy when the beneficiary of a change is the same person who can make it — in a program, I am my stakeholder, I define “useful.” In Software, Optimizing is super hard because the people benefiting from the software and the people who can change it are distant in the communication graph. The feedback loop is long and circuitous.

On one hand, we can work on bringing developers closer to people who can measure benefits. On the other hand, can we find a place where this is already the case?

In development teams! Developers automating for their own team is low effort, high payoff. Personally, I can make programs that only benefit me, which is limited. Go one level up: how much more work is it to write code that benefits my team? If we automate what we already do, then: it only has to run on a few known computers, no one is forced to use it, everyone who uses it can change it so tests are optional. This gives me some scale of value, with little scale in cost. If it turns out really useful, other teams in the org can copy it.


I wish I had automation to set up a build on Travis. A minimum value is certain: I want it right now. The cost is uncertain, but discoverable. I can risk spending one 20-minute iteration on it. At the end of that iteration, I’ll have more information about the cost. I can give it another 20 minutes, or set the idea aside until I hit the same problem again, which gives me information about the value. My pair and I (or the mob in mob programming) become a microcosm of a product team, for whatever timebox we set.

It helps, so do it more.

At some point we automated tests, installed version control, set up continuous integration. These are automations we no longer do without, although at one point we didn’t know how useful they’d be. What will be the next automation we won’t work without?

Here are some examples of local or personal automation that I heard about or used today.

As developers we have this superpower of automation. I think we should use it more on ourselves. We will learn more about our own work by studying how to improve it. And we will improve it, and continually grow our ability to create reliable, persistent, user-experienceable, compatible Software.

Write shell scripts! and Slack bots! and Jenkins plugins! At Atomist I have a platform to more easily scale up my automations from local to team; we’re building it. Don’t wait for us, though. Local automation is a sweet spot no matter what tool you use. Accelerate yourself.

The Golden Yak

(This post continues from the Royal Yak, and concludes the series A Taxonomy of Yaks.)

When we improve how we work, we make tasks faster. We make progress smoother. This is magnified when we improve how all our team members work, or our whole community. Now and then, though, an improvement turns into something more: a real change. This is what I’m aiming for when I shave a Golden Yak.

Golden Yak

The Golden Yak is the yak that changes everything, the one that alters your reality. Sometimes when you follow a problem all the way down — how do I make this easier? … now how do I make that easier? … now what if this were no work at all? you find a different universe. Making something sufficiently easier can produce a sea change, which changes our behavior.

quotes from yakbreeder.com

Example: git. It changed my behavior. Git gives me logs at my fingertips, where they used to be behind a network connection so I never used them. And it gives me a capability I didn’t have before: separating saving my work from sharing my work. The combination is magic; now I save my work carefully with a story and later read that story.

Continuous Integration. This was a huge yak for the people who first did it. It’s still a lot of work, but we wouldn’t live without it, because we (as an industry) have learned that it smooths our future work. It changes our universe by eliminating the tradeoff between frequent vs safe releases. Automated tests also change our work. They reduce fear, and that’s huge. Tests turn canyons into potholes.

Golden yaks change behavior. They start by making something so much easier that we do it way more, then they give us an option we didn’t have before or eliminate a tradeoff.

Here’s a tiny one: when I find myself typing the same command a lot, there’s the question: create an alias for this? well, I have an alias set up for “open .bash_profile in an editor” and another for “reload the aliases in .bash_profile”. Creating an alias is so easy now, there’s no reason not to. It’s gone from a tradeoff (is it worth it?) to a “duh.” Like making a commit. Don’t even ask, just do it!

Here’s a team-sized one: automate spinning up a new project. This takes away a tradeoff: “this is really a separate thing, I should make a service for it… but it takes so long, I should glom it on to an existing one.” With the Atomist bot, I made it two slack commands and two in-Slack button clicks to create a repo and a channel and a build and a deployment, and bam I have an empty service in production ready for my new endpoint. It makes creating a new service “duh.” That decision is now all about architecture, not about tradeoffs. This yak is shaved once and for all. (until there’s a change in infrastructure. And then I’ll accommodate that change in one place, instead of various documentations and people’s heads.)

Here’s one that might turn out bigger: Dependency management, unsolved problem. Code reuse, not all it’s cracked up to be. I am against creating dependencies for code that happens to be the same, that isn’t business-crucial-same. So, what’s the alternative? Cut and paste, but that’s a lot of work and we often mess it up. 
This is my current golden-yak project: libbits. Maybe this will chew a corner off the nasty problem of dependency management. Atomist has automation infrastructure for modifying code, so I can make a program to do copy, paste, modify for a bit of code that’s handy in multiple projects. And then I can make a program to create that program. I’m following the yak stack, looking for the pot of gold at the end.

Then I’d like to make it easier to make the program that writes a program. Another thing we are making at Atomist is microgrammars, a new way of parsing, somewhere between Regex and a BNF grammar. (I don’t think it counts as a yak when your CEO tells you he wants your help with it.) My personal yak is to put an API on this that is super easy to use, and who knows, we might create a replacement for `sed`.

When are we justified in shaving a Golden Yak? If you don’t work for a development automation company, I don’t know. Getting an automation smooth enough that people use it like breathing is a lot of localized work for a huge but spread out benefit. Still, coding is fun! These yaks make great play.


There’s rarely a direct work-related reason to shave them well enough to find their deep secrets, the surprising wisdom on their shining skin under all that hair. The new way of working that is revealed only after we try it. Change our environment, and thereby change ourselves.

When I sit down to work on a weekend (or some Friday afternoons), I let myself dig into some Trim Yaks, and see how far I can take them. When I do this, I’m studying my own work. As automators, we study our users’ work, right? how they do their jobs. When we automate our own work we learn about our work deeply. Writing code to change code teaches me about code.

Here’s a shortcut to Golden Yak wisdom: cultivate an awareness of what we’re doing while we do it. I learned this trick from a book about how to ride a motorcycle: Devote 5% of your attention to how the work is going, and where it could be smoother. This awareness, then, feeds into our prioritization. And into understanding of our teammates’ experiences.

Conclusion

Yaks are a part of life and of progress.

Pair: We aren’t in this alone, and we can do it better together. Pairing lets us keep perspective on what we’re doing and why. We can use that space to think about doing it better.

Proven usefulness: look for it. Don’t spend a ton of time until you’ve hit a problem a ton of times. Track your yaks and timebox your attacks. Remember it’s about generativity, not all about you. Fill potholes, and shave a little more so your team can share the benefit.

Play: Stay curious. Especially learn about the system you work in, both socially and technically. Learn a little extra each time, about your tools, your system, and your companions on this journey. Interact and automate for the wisdom; for changing us, and thereby the world.

Yaks. They’re so much more than hair.

The Royal Yak

(continued from Trim Yaks; part of the Taxonomy of Yak Shaving series)

Royal Yak, aka Yakkity Yak (quote from yakbreeder.com)

Talking to people is yak shaving; it is an intermediate task that helps you get your official tasks done. It’s usually seen as a separate way to “waste” time. I’m here to call relationship-building out as a productivity booster.
Conway’s Law says that our system architecture resembles our communication flow (approximated by our org chart). Indeed, the human relationship graph has a close relationship to our system relationship graph — and if it doesn’t, that’s going to be a dysfunctional hunk of software.

Those interludes of chit-chat while you’re pairing? They matter. Especially on a remote team, when we don’t have chit-chat in a kitchen. Here are three crucial objectives for talking to people at work:

The more we know about what people know, the more we know whom to ask for help. This is the first thing I ask about when I meet a new person in the company. If my service interacts with another one, it helps to know who understands that one and can expound on its history. If I need to change nginx config, it helps to know that person who likes it.

Asking for help is a beautiful thing: it gets you help, and it gets you friends. Counterintuitively, people like you more if they have helped you, if they feel like you owe them one. This is great, especially with people outside the team, whom you can’t all maintain a close relationship with.

We need to know what our teammates don’t know. In particular, on a close team — and this is why a close team is limited to 7 people — we need to track what each other person knows and doesn’t, so that we can tell them the pieces of information they need. We have our own mental model of the system, our model of each other, and our model of each other’s models — all this is necessary for close coordination. This is how we avoid the potholes of communication failure. This is how we get distributed decisionmaking with trust: because we can predict each others’ actions. For smooth work, we need to know each other this well.

Let our teammates know that we want to know what they know, we want to hear their input. Psychological safety. We each need to feel safe suggesting ideas and asking questions. Google found that this is the #1 determiner of team effectiveness. So part of generativity is making your team a safe place for everyone to contribute. Do this by asking people for input and listening carefully to what they say. We don’t to agree; we have to welcome the information.

Social yaks are yaks. Get to know people, find out what they know, and make them feel safe. Keep them up to date on what you know.

One way I do this is with my own slack channel, #jessitron-stream. I use it as a notebook. I throw error messages in them when I hit them, and record what I did (especially when changing infrastructure). I track my own yak stack. People can pop in and answer questions if they want, or I can @ them and then they see the context of my inquiry. Lots of other Atomists have started their own stream channel; we have #radio-russ and #kipz-corner and #sylvain-stream and #hoffhaus and #arrrrr-day. It’s one way to work in public from the privacy of our own homes.

Don’t neglect the Royal Yaks. Hold court in Slack or in the kitchen. Interact with each other. Find out what is useful and what is in the way of other people, and you can prioritize your own yak shaving. These conversations, when they get interesting, might even lead you to the Golden Yak (ooooo).

Trim Yaks

(continued from Imperial Yaks; part of A Taxonomy of Yak Shaving series)

Trim Yaks, aka, the Hackhacking yak (making coding faster)

Each of the previous yaks stood in the way of a particular task. The Trim Yak is not so task-specific; these are the ones that let us work faster generally.

I nickname them “Hackhacking Yaks” after Lifehacking, which includes all the ways we change our environment in order to influence our own behavior. Change your world in order to change yourself, the feedback loop of human progress. Here, we’re talking about writing or configuring programs in order to change the way we code: Hackhacking. (It’s fun to say.)

At one extreme: how long between receiving your work laptop and when you can get work done smoothly? At least a couple days. There are essential Hackhacking yaks to shave here. Gotta install all the tools, gotta set up all the credentials. But then there’s more. Dotfiles, IDE configuration. For me, I cannot communicate in Slack until I’ve imported a few dozen essential emoji. To do that, I install a Chrome plugin and log in to Dropbox — Trim Yaks can stack up too.

These yaks may look like procrastination. And they can be, let run loose; configuring a keyboard shortcut can turn into writing a vim plugin.
For me, it’s stuff like, I have a library that’s used by another library that’s used by a service I want to test, which is a lot of installing and version-updating locally and doing that manually is just wrong and I want to automate it all! Or, something is wrong with my git repo, it’s in some weird state and I want to know how it got there and can I go read man pages until I understand it. Or, I hate changing nginx configuration because I don’t understand it, can I spend a day learning nginx configuration?

All of these things are useful. And they’re the kind of thing that when I do them, they pay off in ways I never would have expected. Automating the repository linkages (which I did on a weekend, for play) left me with a little toy program that later when I picked up Neo4J it was fast to implement a piece that makes a little dependency graph. Early in my git usage (this was 2012) I spent 6 hours (at work) watching a few videos and reading the book until I grasped the concepts. Later spent 45 minutes writing a talk that launched my speaking career. I still haven’t learned nginx configuration but gah! I so wish I had.
Trim yaks are about increasing our speed over time. It’s hard to know how useful a given Trim Yak will be; we might never use it again or we might use it a million times. Benefit is uncertain, so limit the cost.

Here’s a strategy:

Timebox! The first time something bugs you (“gosh, this code change could be faster, I bet the IDE has tools for this”), do it by hand. The second time, think about changing it: can you give it 15 min? that might be enough. If it isn’t, then the third or fourth time, give it an hour. Spend time only on what has proven useful.

And more considerations:

Stay curious! Learn a little bit more than you need to. I need one nginx option, I’ll look past the first StackOverflow answer, check the docs, and get a clue about why it works, or learn some new vocabulary word.

Pairing: I find pairing a big help with the Imperial Yaks; it helps to have someone else who is like “yeah we really have to fix this.” But it’s hard to shave Trim Yaks while pairing. Except! Pairing is a great time to share trimming tips. “I wish I had a faster way to do this, do you know one?” When your partner can help you get that IDE plugin set up, it’s efficient and also builds relationships.

Here’s the one Trim Yak that will pay off more than others: learning the system you work on. This is the one trick that separates hyperproductive developers from struggling ones: intimate knowledge of the system we work on. And it’s very, very hard to achieve in a system we didn’t write ourselves. Build your mental model at every opportunity. This yak that pays off most and fastest.

When pairing (or mobbing), learn what your partner (or everyone) knows.

When coding alone, dig around. Write tests, or break tests, or refactor. If you refactor, be careful: you built up your own mental model, your brain’s entwinement with the code, but you’ve invalidated the mental model other people have. Consider throwing the refactor away. Sometimes the team is like “this is a mess please change it,” but other times you’re not so much shaving a yak as combing the hair in a different direction.

Extract more value: Trim yaks increase personal productivity. If we take them a little farther, they can be generative. Post useful scripts or shortcuts in Slack. Document how to develop and test. If you do enjoy nginx configuration, tell people that you’re interested in it. They’ll be happy to share their nginx yaks with you next time, and you can build that skill. Blog! Those little tricks that you don’t use very often, let Google find them for you next time, and save hundreds of developers from your pain.

Share what you learn, and find out what’s holding other people back. You can do that with the next yak… the Royal Yak!

The Imperial Yaks

(continued from Attack Yak; series begins with Taxonomy of Yak Shaving) Sometimes you’re coding along, writing tests as little experiments “this should fail because I haven’t implemented the parser for it yet” — and it fails in a way you didn’t expect. And then you start digging and the parsing library isn’t working how you expected. And that library (it’s our library) doesn’t have a test for this case. Now you need to switch your development environment over. The yak stack is getting deeper. You have encountered …

The Imperial Yaks!

Imperial Yaks, aka the Yak Stack; quote from yakbreeder.com

These are the yaks that stack on top of each other. They all deserve to be shaved. This is a real bug in the library. It needs more tests. Then fix the bug, then deal with the linter and make a PR and wait for its review and do a release… and in the meantime to move forward we must figure out how to get npm to use local dependencies, which is not trivial with TypeScript.

How far do you go? When do you look for a workaround — maybe we can parse this with regular expressions and not use our library. When do you ask “Is this change even worth making?”

Pairing helps. I love pairing because I can be down in the weeds of the syntax and the particular bug while my pair still has part of her brain operating at the level of “why are we doing this again?”

If your original task is urgent: look for workarounds sooner. Never abandon the yak silently; create an issue if you don’t have time to fix it. Leave enough information to reproduce the bug. (This is work.) At the very least, throw something into Slack for tracking.

If the yaks you uncovered are dangerous: they might be more important that your original task. Realize you’re doing something important. Update your team and re-orient.

In the example about that failing test, it’s Rod who hit that last Saturday. He didn’t get his particular goal accomplished over the weekend. But he fixed a bug that was going to hold Clay back. He added tests. I popped in and paired on it, and added a few more pending tests for corner cases, which Rod later fixed. Sylvain merged the changes and did a release. All of us are more concerned with Atomist as a whole than whatever assigned tasks are on our list. That’s our culture.

When I aim for generativity, I can think more broadly about yaks. I can decide whether the yak stack confronting me is worth defeating, or going around. I feel good about creating an issue, if that’s all I can do right now: a warning sign to other people. Or leaving the code a little clearer than I found it: more null checks, an extra test, type declarations (in gradually-typed languages like TypeScript or Clojure).

My yak stack is now deeper than it used to be. Every confusing error I get, I solving it until I’ve made the error more communicative, seen the new message, and then committed that. Or changed the README. When I started at Stripe, after 4 months I felt like I wasn’t getting enough work done, and when I expressed this to others — because when I’m freaking out, it’s my job to ask for help — Julia and Danielle were like, but you’ve been making so many commits! Some managers at some companies look only at assigned tasks, but you can bet all my coworkers saw those README and error message and infrastructure changes and thought “Thank you!” about each one. Stripe has a culture of generativity, so I was fine.

It isn’t just about me and my personal productivity. Teamwork isn’t a race, with each person trying to be faster than the rest of the team.

Yak Racing

In the past, I’ve worked with developers who hoarded information, excluding others from customer meetings, insisting on a code structure that makes sense to them and no one else, turning work into a competition with the rest of us.

This exclusively-personal-maximization is also a danger of the next kind of yak: the Trim Yak.

The Attack Yak

(This post describes the first yak category. You might choose to start with the Taxonomy of Yak Shaving intro.)

Black Yak, aka Attack Yak. (quotes are from yakbreeder.com)

When you feel that your task is 80% complete, these yaks form the next 80%. They are the distance between “works on my box” and “this is done.”
All the data setup. Verification. Approvals. Status checks. Linters and code review. These are the yaks we have summoned voluntarily. There are also surprises like, I’m ready to integration test, but the dev environment is broken. Or merge conflicts. Or an unrelated test failure. Or someone wants to argue about whether this change is even a good idea.

Attack yaks are very frustrating, but there’s no getting around them (without cheating). You have to shave them. They make our development flow rough, by forcing us to context switch. What can we do?

Be cautious about summoning these yaks. We choose to all write automated tests. It doesn’t make us faster today, but it makes our development smoother in the medium and long term. Except when it doesn’t; intermittent failures, especially in tests that are too wide to be diagnostic, can make development rough instead. Many fine-grained tests prevent refactoring, or enforce functionality that isn’t needed anymore but nobody knows it. Be conscious in your yak shaving. Don’t shave its eyebrows.

On some projects, we summon the yak of code reviews. What is the objective? If it’s code quality, pairing is live, yakless code review with no context switches yeah! If it’s keeping up the team’s mental model, then keep reviews focused on that; discourage nitpicks. As a reviewer, if you want to see another test, add it. Give me a commit instead of an attack yak.

:attack-yak:

Track the attacks. We have a yak emoji in Slack for these.

:attack-yak: Staging environment is down, I can't test

If we all do that, we can search Slack before the retrospective to find out which ones are interrupting us the most. Our retro can have a little more data than our gut feel, which is really based on the last few days.

Coordinate your attack. Once we know which yaks are holding us back, shave them with automation. Shave them until they’re not an interruption anymore. Autofix on the linter. Script creation of the dev environment, so that “it’s broken” means “recreate it” instead of “figure out what the problem is today and create a special custom fix.” Then automate regular re-creation.

This work can feel like a waste of time, because it isn’t getting features delivered. Yet, if we think about our work as more than personal productivity, we see the value. On my team, we care about generativity (as opposed to productivity): the difference in what your team does with you, vs what it would get done without you. It’s about the team, not about me. And it’s about the team in the future, not only right now. Not about going faster, but about going smoother: removing obstacles for everyone, present and future us.

Every time you hit a build or environment failure, ask yourself, “How could this be easier to diagnose?” It might be “put a better error message here.” Fill the potholes you encounter. Smooth the path for future-you and future-team.

Sometimes you’re coding along, writing tests as little experiments “this should fail because I haven’t implemented the parser for it yet” — and it fails in a way you didn’t expect. And then you start digging and the parsing library isn’t working how you expected. And that library (it’s our library) doesn’t have a test for this case. Now you need to switch your development environment over to work on that (have you automated that?). The yak stack is getting deeper. You have encountered …

The Imperial Yaks!