The artifact and the process

again: Every action has two results: a set of side effects on the world, and the next version of ourselves.

Sometimes in the creation of an artifact, the artifact itself is less important than the process of creating it. Wardley Maps are like that too – it’s about the thought that goes into creating them, and the conversations they enable. (video by @catswetel)

Most diagrams, especially on the white board, communicate very well to the people who participated in their creation, and very little to anyone else.

It’s the co-creation that matters. The participation in creating the artifact. The output is the next version of our team, with this common ground we established in the process.

Action is a process. Reasons matter.

Knowledge work has to be done for the right reason to be done well.

If I choose to implement a feature because I’ll get some reward for it, like a gold star on my next review, then I will implement it far enough to check the box. I won’t get it smooth, or optimal. I will optimize the wrong parts or not at all.

If I implement that feature because I believe it’s the next thing I can do to improve our customer’s experience, then I’ll get optimize for that experience. I’ll make all the tiny decisions with that in mind. I’ll check whether I’ve slowed the site down, or interfered with other features, or cluttered the interface.

Creation is a process, not a transaction.

We are guided throughout that process by an intention. If that intention is to check a box, we’ll check the box, and leave the rest of the system to fall as it will.

In Dynamics in Action, Alicia Juarrero models human action as a process: we start with an intention, which shapes the probability space of what we do next. We do something, and then we check the results against our intention, and then we respond to difference.

Action is a process, not an atomic motion.

Development is action, and it is knowledge work not hand work. To do it well, to shape the system we work in and work on, we line our intentions up with the goals of the system. Not with some external reward or punishment. Intention shapes action, over time, in response to feedback. Reasons matter.

Deliberate impermanence

There’s something magical about post-its falling off walls.

ancient post-its on my own wall

I think about this as my team spends time pruning tickets, closing ones that seemed important at the time, but now aren’t worth doing. These are ideas that need to fall through the cracks, to slide down the wall behind the file cabinet, to let us focus on the few that matter.

Instead we’re spending time and decisionpower on eliminating old ideas. Me, I’d rather dump the lot of them and resurrect the crucial few.

Then consider the connections between tasks: Christian says that before we create this command form, we need the http endpoint story completed. I could set up a relationship between these tickets, we could make a whole diagram and tie our current priorities into a line. Yet, interdependencies change and shift. We switch priorities quickly as we learn about customers.

The tenacious task tracker is stronger that sticky notes. At best, it’s out of date. At worst, it leads to ossification, to attachment to old plans. If it’s too hard to change this tracking…

Wait. I think it’s more than this. “Easy to change” isn’t enough. “Trivial to change” isn’t enough. The post-its fall on the floor, we pick them up, we put them back, “Oh actually this belongs over here. This one doesn’t even matter anymore. Ah, that was done weeks ago.” The wall of stickies requires active maintenance even to stay the same. It naturally degrades. This leaves room for evolution. Every post-it on the floor is an opportunity for a new start.

So we don’t set up dependencies between tasks. We leave those in our heads, or individual notebooks. I put them on seamaps sometimes, which I use to summarize the daily standup; these only last a day.

a standup seamap: bunch of task mountains with lines to each other and the star-goal; people in boats headed toward mountains.

Embrace degradation and dump those piles of precious, stale ideas. Be cautious about weaving tight plans and documenting them: persistence has a hidden price.

Free idea: chaos gnome for task trackers?

Hyperproductive development

TL;DR: the most productive development happens when one person knows the system intimately because they wrote it; this is in conflict with growing a system beyond what one person maintains.

Let’s talk about why some developers, in some situations, are ten times more productive than others.

hint: it isn’t the developers, so much as the situation.

When do we get that exhilarating feeling of hyperproductivity, when new features flow out of our fingertips? It happens when we know our tools like the back of our hands, and more crucially, when we know the systems we are changing. Know them intimately, like I know the contents of my backpack, when I packed it and I tuned the items in each pouch over years of travel. Know the contents of every module, both what they are and what we’d like them to be if we ever finish that refactoring. Know the edges, who uses every API and which changes will break whom, and we’re friends with all of the stakeholders. Know the underpinnings, which database fields are indexed and which are obsolete and which have quirky special values. Know the infrastructure, where it runs in production and how to ssh in; where it runs in test and what version is deployed and when it is safe to push a new one. Know the output, what looks normal in the logs and what’s a clue. We have scripts, one-liners that tail the logs in all three prod instances to our terminals so our magic eyes can spot the anomaly.

We know it because we wrote it, typically. It is extremely difficult to establish this level of intimacy with an existing system. Braitenberg calls this the Law of Downhill Invention, Uphill Analysis. Complex systems are easier to build than to figure out after they’re working.

We know it because we are changing it. The system is alive in our head. It’s a kind of symbiosis: we help the system run and grow, and the system works the way we wish. If we walk away for a month or two, begin a relationship with a different system, the magic is lost. It takes time to re-establish familiarity.

Except, I’m suspicious of this description. “We” is a plural pronoun. This depth of familiarity and comfort with a system is personal: it’s usually one person. The one person who has conceived this solution, who holds in their head both the current state and where they are aiming.

If you are this person, please realize that no one else experiences this work the way you do. For other people, every change is scary, because they don’t know what effect it will have. They spend hours forming theories about how a piece works, and then struggle to confirm this with experiment; they don’t have the testing setup you do. They study every log message instead of skimming over the irrelevant ones, the ones you’ve skipped over so often you don’t even see them anymore. By the time they do figure something out, you’ve changed it; they can’t gain comprehension of the system as quickly as you can alter it. When they do make a change, they spend lots of time limiting the scope of it, because they don’t know which changes will cause problems. They get it wrong, because they don’t know the users personally; communication is hard.

If you are this person, please go easy on everyone else. You are a synthetic biologist and can alter the DNA of the system from within; they are xenosurgeons and have to cut in through the skin and try not to damage unfamiliar organs.

If you work with this person, I’m sorry. This is a tough position to be in, to always feel inferior and like you’re breaking everything you touch. I’m there now, in some parts of our system. It’s okay for me because the host symbiont, the author and manipulator of that software, is super nice and helpful. He doesn’t expect me to work with it the same way he does.

If your team looks like this, here are some steps to take:

  1. Consider: don’t change it. This really is the fastest way to develop software. One person, coordinating with no other developers, can move faster than a whole team when the system is small enough. Until! we need the system to grow bigger. Or! the system is crucial to the business (it’s an unacceptable risk for only one person to have power over it).
  2. As the host symbiont who lives and breathes the system: strike the words “just”, “easy,” “obvious,” “simple,” and “straightforward” from your vocabulary. These words are contextual, and no other human shares your context.
  3. Please write tests. Tests give people who are afraid of unintentional breakages a way to test their theories. Experimentation is crucial to learning how a system works, and tests make experiments possible. They also serve as documentation of intended behavior.
  4. Pair program! By far the best way to transfer understanding of the system to another human is to change it together. (Or boost a whole team at once: mob program!)
  5. Make a README for other developers. Describe the purpose of the system briefly, and document how you develop, test, and troubleshoot the system. Specify the command lines for running tests, for deployment, for accessing logs. Describe in detail how to obtain the necessary passwords. Write down all the environments where it runs, and the protocol around changing them.
  6. Do you know your users better than anyone else? Remedy that. Bring other team members into the discussion. (There’s a sweet spot of a single developer-type who works within a business unit. When the software becomes too important for this scale, it gets harder.) Let all the devs get to know the users. Have happy hours. Form redundant communication channels. It’ll pay off in ways you never detect.
  7. Slow down. Like seriously, if one person is developing at maximum speed on a project, no one else can get traction. You can’t move at full speed and also add symbionts. When it is important to bring in new people, don’t do anything alone. Pair on everything. Yes, this will slow you down. It will speed them up. Net, we’ll still be slower than you working alone. This is an inherent property of the larger system, which now includes interhuman coordination. There’s more overhead than when it was just you and your program. That’s OK; it’s a tradeoff for safety and scale from sheer speed.

Let’s acknowledge that there really are developer+situations that are 10x more productive than others. Let’s acknowledge that they don’t scale. Make choices about when we can take advantage of the sweet spot of local or individual automation, and when the software we’re building is too important for a bus factor of one.
Distinguish between an experimental prototype, when speed of change and redirection is crucial, versus a production app which needs backwards compatibility guarantees and documentation and all that seriousness — this requires a solid team.

Recognize that the most productive circumstance for development is a rare circumstance. Most of the time, I need to work on a system that someone else wrote. (that “someone else” could be “me, months or years ago.”) The temptation to rewrite is strong, because if I rewrite it then I’ll understand it.

There’s a time for 10x development, and a time for team development. When you want to be serious, the 10x developer prevents this. If that’s your situation, please consider the suggestions in this post.

Today’s Rug: maven executable jar

I like being a polyglot developer, even though it’s painful sometimes. I use lots of languages, and in every one I have to look stuff up. That costs me time and concentration.

Yesterday I wanted to promote my locally-useful project from “I can run it in the IDE” to “I can run it at the command line.” It’s a Scala project built in maven, so I need an executable jar. I’ve looked this up and figured this out at least twice before. There’s a maven plugin you have to add, and then I have to remember how to run an executable jar, and put that in a script. All this feels like busywork.

What’s more satisfying than cut-and-pasting into my pom.xml and writing another script? Automating these! So I wrote a Rug editor. Rug editors are code that changes code. There’s a Pom type in Rug already, with a method for adding a build plugin, so I cut and paste the example from the internet into my Rug. Then I fill in the main class; that’s the only thing that changes from project to project so it’s a parameter to my editor. Then I make a script that calls the jar. (The script isn’t executable. I submitted an issue in Rug to add that function.) The editor prints out little instructions for me, too.

$ rug edit -lC ~/code/scala/org-dep-graph MakeExecutableJar main_class=com.jessitron.jessMakesAPicture.MakeAPicture

Resolving dependencies for jessitron:scattered-rugs:0.1.0 ← local completed
Loading jessitron:scattered-rugs:0.1.0 ← local into runtime completed
run `mvn package` to create an executable jar
Find a run script in your project’s bin directory. You’ll have to make it executable yourself, sorry.
Running editor MakeExecutableJar of jessitron:scattered-rugs:0.1.0 ← local completed

→ Project
  ~/code/scala/org-dep-graph/ (8 mb in 252 files)

→ Changes
  ├── pom.xml updated 2 kb
  ├── pom.xml updated 2 kb
  ├── bin/run created 570 bytes
  └── .atomist.yml created 702 bytes

Successfully edited project org-dep-graph

It took a few iterations to get it working, probably half an hour more than doing the task manually.
It feels better to do something permanently than to do it again.

Encoded in this editor is knowledge:
* what is that maven plugin that makes an executable jar? [1]
* how do I add it to the pom? [2]
* what’s the maven command to build it? [3]
* how do I get it to name the jar something consistent? [4]
* how do I run an executable jar? [5]
* how do I find the jar in a relative directory from the script? [6]
* how do I get that right even when I call the script from a symlink? [7]

It’s like saving my work, except it’s saving the work instead of the results of the work. This is going to make my brain scale to more languages and build tools.

——————————————–
below the fold: the annotated editor. source here, instructions here in case you want to use it -> or better, change it -> or even better, make your own.

@description “teaches a maven project how to make an executablejar”
@tag “maven”
editor MakeExecutableJar

@displayName “Main Class”
@description “Fully qualified Java classname”
@minLength 1
@maxLength 100
param main_class: ^.*$

let pluginContents = “””
        org.apache.maven.plugins
        maven-shade-plugin
        2.4.3
       
          target/executable.jar [4]       
       
         
            package
           
              shade
           
           
             
               
                  __I_AM_THE_MAIN__
               
             
           
         
       
     
“”” [2]

let runScript = “””#!/bin/bash

# http://stackoverflow.com/questions/59895/getting-the-source-directory-of-a-bash-script-from-within
SOURCE=”${BASH_SOURCE[0]}” 
while [ -h “$SOURCE” ]; do # resolve $SOURCE until the file is no longer a symlink
  DIR=”$( cd -P “$( dirname “$SOURCE” )” && pwd )”
  SOURCE=”$(readlink “$SOURCE”)”
  [[ $SOURCE != /* ]] && SOURCE=”$DIR/$SOURCE” # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
done [7]
DIR=”$( cd -P “$( dirname “$SOURCE” )” && pwd )” [6]

java -jar $DIR/../target/executable.jar “$@” [5]
“””

with Pom p
  do addOrReplaceBuildPlugin “org.apache.maven.plugins” “maven-shade-plugin” pluginContents [1]

with File f when path = “pom.xml” begin
  do replace “__I_AM_THE_MAIN__” main_class
  do eval { print(“run `mvn package` to create an executable jar”)[3]
end

with Project p begin
  do eval { print(“Find a run script in your project’s bin directory. You’ll have to make it executable yourself, sorry”) }
  do addFile “bin/run” runScript
end

It’s Atomist Time!

I’m hella excited to get to work on Atomist full-time starting now (January 2017). Why? What do they do? Oh let me tell you!
I love developing software, not least because we (as an industry) have not yet figured out how to software. We know it’s powerful, but not yet how powerful. Software is like engineering except that the constraints aren’t in what we can build, but what we can design and specify. Atomist is expanding this capacity.
Atomist builds tooling to smooth some bumps in software development. There are three components that I’m excited about, three components that open new options in how we develop software.

Component 1: Code that changes code

First, there are code editors, called Rugs. On the surface, these automate the typing part. Like code generators, except they continue to work with the code after you modify it. Like refactorings in an IDE, except they appear as a pull request, and then you can continue development on that branch. If you have some consistent code structure (and if you use a framework, you do), Rugs can perform common feature-adding or upgrading or refactoring operations. Use standard Rugs to, say, add graph database support to an existing Spring Boot project. Customize Rugs to set up your Travis build uniformly in your projects. Create your own Rugs to implement metrics integration according to your company’s standards — and to upgrade existing code when those standards change.
On the surface this is an incremental improvement over existing code generation and IDE refactoring tools. Yet, I see it as something more. I see it as a whole new answer to the question of “indirection or repetition?” in code. Take for instance: adding a field to a Rails app makes us change the controller, the model, and four other places. Or creating a new service means changing deployment configuration, provisioning, and service discovery. Whenever a single conceptual change requires code changes in multiple spots, we complain about the work and we make mistakes. Then we start to get clever with it: we introduce some form of indirection that localizes that change to one place. Configuration files get generated in the build, Ruby metaprogramming introduces syntax that I can’t even figure out how it’s executable — magic happens. The code gets less explicit, so that we can enforce consistency and make changing it … well, I’m not gonna say “easier” because learning to cast the spell is tricky, but it is less typing.
Atomist introduces a third alternative: express that single intention (“create a new service” or “add this field”) as a Rug editor. This makes writing it one step, and then the editor makes all those code changes in a single commit in a branch. From there, customize your field or your new service; each commit that you make shows how your feature is special. The code remains explicit, without additional magic. When I come back and read it, I have some hope of understanding what it’s doing. When I realize that I forgot something (“oops! I also need to add that service to the list of log sources”) then I fix it once, in the NewService.rug editor. Now I never forget, and I never have to remember.
I love this about developing with Rugs: as I code, I’m asking myself, “how could I automate this?” and then what I learn is encoded in the Rug, for the benefit of future-me and (if I publish it) of future-everyone-else. That is when I feel productive.

Component 2: Coordination between projects

Editors are cute when applied to one project. When applied across an organization, they start to look seriously useful. Imagine: A library released a security update, and we need to upgrade it across the organization. Atomist creates a pull request on every project that uses that library. The build runs, maybe we even auto-merge it when the build passes. Or perhaps there are breaking changes; the editor can sometimes be taught how to make those changes in our code.
And if a Rug can change the way we use a library, then it can change the way we use ours. This is cross-repository refactoring: I publish an internal library, and I want to rename this function in the next version. Here’s my game: I publish not only the new version of my library, but an editor – and then I ask Atomist to create pull requests across the organization. Now it is a quick code review and “accept” for teams to upgrade to the new version.
Atomist coordinates with teams in GitHub and in Slack. Ask Atomist in Slack to start that new feature for you, or to check all repositories in the organization and create pull requests. Atomist can also coordinate with continuous integration. It ties these pieces together across repositories, and including humans. It can react to issues, to build results, to merges; and it can ping you in Slack if it needs more information to act appropriately. I have plans to use this functionality to link libraries to the services that use them: when the build passes on my branch, go build the app that uses my library with this new version, and tell me whether those tests pass.
This is cross-repository refactoring and cross-repository build coordination. This gives companies an alternative to the monorepo, to loading all their libraries and services into one giant repository in order to test them together. The monorepo is a lie: our deployments are heterogenous, so while the monorepo is like “look at this lovely snapshot of a bunch of code that works together” the production environment is something different. The monorepo is also painful because git gets slow when the repository gets large; because it’s hard to tell which commits affect which deployed units; and because application owners lose control over when library upgrades are integrated. Atomist will provide a layer on top of many repositories, letting us coordinate change while our repositories reflect production realities.
Atomist tooling will make multirepo development grow with our codebases.

Component 3: is still a secret

I’m not sure I can talk about the third piece of possibility-expanding tooling yet. So have this instead:
Automated coordination among systems and people who interact with code — this is useful everywhere, but it’s a lot of work to create our own bots for this. Some companies put the resources into creating enough automation for their own needs. No one business-software-building organization has a reason to develop, refine, and publish a general solution for this kind of development-process automation. Atomist does.
When it becomes easy for any developer to script this coordination and the reactions just happen — “Tell me when an issue I reported was closed” “Create a new issue for this commit and then mark it closed as soon as this branch is merged” — then we can all find breakages earlier and we can all keep good records. This automates my work at a higher level than coding. This way whenever I feel annoyed by writing a status report, or when I forget to update the version in one place to match the version in another, my job is not to add an item to a checklist. My job is to create an Atomist handler script to make that happen with no attention from me.

My secret

I love shaving yaks. Shaving them deeply, tenderly, finding the hidden wisdom under their hair. I love adding a useful feature, and then asking “How could that be easier?” and then “How could making that easier be easier?” This is Atomist’s level of meta: We are making software to make it easier for you to make your work easier, as you work to make software to make your customers’ lives easier.
I think we’re doing this in depths and ways other development tools don’t approach. At this level of meta (software for building software for building software for doing work), there’s a lot of leverage, a lot of potential. This level of meta is where orders-of-magnitude changes happen. Software changes the world. I want to be part of changing the software world again, so we can change the real world even faster.
With Atomist, I get to design and specify my own reality, the reality of my team’s work. (Atomist does the operations bit.) Without spending tons of time on it! Well, I get to spend tons of time on it because I get to work for Atomist, because that’s my thing. But you don’t have to spend tons of time on it! You get to specify what you want to happen, in the simplest language we can devise.
We’re looking for teams to work with us on alpha-testing, if you’re interested now. (join our slack, or email me) Let’s learn together the next level of productivity and focus in software development.