Work to Rule

There’s a particular kind of almost-strike called “work-to-rule.” It’s when workers in a factory “adhere strictly to all the working rules laid down by their employers, with the deliberate intention of reducing the rate of working.” Doing exactly what they’re told, and nothing else, means production grinds to a halt. Every formal system in life “is parasitic on an informal system.”[2] We count on people to notice problems, to adjust and improvise in small ways. 

The perfect example of work-to-rule is: every program ever. The computer always does exactly what it’s told, nothing less or more. This is incredibly hard to get right! Every human endeavor so far, we rely on people with local awareness. Your blueprint may be lovely, but it’s a construction worker who’s going to notice when the materials aren’t quite right and the walls aren’t fitting, and react to it. A person on the manufacturing line will see when the product is out of whack and correct it. They learned what “out of whack” means from someone else. There’s an informal system underneath.

In the program, there is no person on the ground when decisions are made at runtime. All of that awareness, everything that could go wrong and should be noticed, it all must be programmed in up front. That pushes all of the human context up a level: it’s the developers who supply all of it. We need to understand the program’s environment, the business context, the data, everything. Which problems matter? What can we fix, and what should halt the whole operation? What clues can tell it “this is out of whack?” How will our program react to surprise?

This means all the creativity and conscientiousness and context of all the runtimes is crammed into the development at code-time. The team communication, the adjustment, the humanity: maximize this on the development team. Because we need every ounce of this we can get of this going into the code. It’s all we have to lean on.

We’ve seen from the waterfall process what happens when we try to create a program based on a written set of requirements. Create a development process with a bunch of rules and regulations, emphasizing vertical (in the org chart) communication, and what will you get? That program is already work-to-rule. When the people coding it are also working-to-rule, to any degree, then the code is brittle and unresponsive, just like the production output of a plant whose workers are on this almost-strike.

Be careful about the rules we impose on development. Communication YES, knowledge sharing YES. But let the rules be the ones the work itself imposes. Blind adherence to standards, that’s what the computers are for.

—————

[1] http://www.thefreedictionary.com/work-to-rule

[2] Seeing Like a State, James C Scott. Introduction.

Work To Rule

There’s a particular kind of almost-strike called “work-to-rule.” It’s when workers in a factory “adhere strictly to all the working rules laid down by their employers, with the deliberate intention of reducing the rate of working.” Doing exactly what they’re told, and nothing else, means production grinds to a halt. Every formal system in life “is parasitic on an informal system.”[2] We count on people to notice problems, to adjust and improvise in small ways. 

The perfect example of work-to-rule is: every program ever. The computer always does exactly what it’s told, nothing less or more. This is incredibly hard to get right! Every human endeavor so far, we rely on people with local awareness. Your blueprint may be lovely, but it’s a construction worker who’s going to notice when the materials aren’t quite right and the walls aren’t fitting, and react to it. A person on the manufacturing line will see when the product is out of whack and correct it. They learned what “out of whack” means from someone else. There’s an informal system underneath.

In the program, there is no person on the ground when decisions are made at runtime. All of that awareness, everything that could go wrong and should be noticed, it all must be programmed in up front. That pushes all of the human context up a level: it’s the developers who supply all of it. We need to understand the program’s environment, the business context, the data, everything. Which problems matter? What can we fix, and what should halt the whole operation? What clues can tell it “this is out of whack?” How will our program react to surprise?

This means all the creativity and conscientiousness and context of all the runtimes is crammed into the development at code-time. The team communication, the adjustment, the humanity: maximize this on the development team. Because we need every ounce of this we can get of this going into the code. It’s all we have to lean on.

We’ve seen from the waterfall process what happens when we try to create a program based on a written set of requirements. Create a development process with a bunch of rules and regulations, emphasizing vertical (in the org chart) communication, and what will you get? That program is already work-to-rule. When the people coding it are also working-to-rule, to any degree, then the code is brittle and unresponsive, just like the production output of a plant whose workers are on this almost-strike.

Be careful about the rules we impose on development. Communication YES, knowledge sharing YES. But let the rules be the ones the work itself imposes. Blind adherence to standards, that’s what the computers are for.

—————

[1] http://www.thefreedictionary.com/work-to-rule

[2] Seeing Like a State, James C Scott. Introduction.

Tools

There are these wineglasses that my partner likes. They’re pretty, but they’re so tall they only fit in the cupboard above the fridge. He can reach that cupboard, but I have to fetch a stool to get in there. He uses these glasses all the time, while I always choose the squattier glasses that I can reach unaided. It’s free (in time costs) for him to use the pretty glasses, expensive for me.

At work, what tools do you reach for that your coworkers don’t? Or do some of them have tools that you ask them to use for you, because they understand them and you’d have to look stuff up?

When something is free, we use it all the time. Once we do that, we get benefits never thought possible. Branching in git is like that. Autocomplete as a way of exploring an API. Some of us use grep and cut to explore log files, because we know these tools well enough that it’s just as easy as opening the file and searching inside. At a previous job, I was the fastest at production support because I built up a library of SQL joins that tracked down the useful data.

When we take the time to learn a new tool, it’s like we get taller. We can bring down answers and build solutions that were out of reach before. That’s not an option for me with the wine glasses, but it is with Chef and Jenkins, Graphite and the Typesafe Console, Kleisli and Liebniz (strange scalaz types).

Learning a new tool might feel like yak-shaving. Learn it well enough that it’s free (no cognitive load), and you just gained a few inches on everyone else.

Weakness and Vulnerability

In computers and in people, weakness and vulnerability are different. Separate the concerns: [1]

Vulnerability is an openness to being wounded.

Weakness is inability to live through wounds.

In D&D terms: vulnerability is a low armor class, weakness is low hit points. Armor class determines how hard it is for an enemy to hit you, and hit points determine how many hits you can take. So you have a choice: prevent hits, or endure more hits.

If you try to make your software perfect, so that it never experiences a failure, that’s a high armor class. That’s aiming for invulnerability.

Thing is, in D&D, no matter how high your armor class, if the enemy makes a perfect roll (a 20 on a d20, a twenty-sided die), that’s a critical hit and it strikes you. Even if your software is bug-free, hardware goes down or misbehaves.

If you’ve spent all your energy on armor class and little on hit points, that single hit can kill you.

Embracing failure is about letting go of ideal invulnerability, and thinking about recovery instead. I could implement signal handlers, and maintain them, and this is a huge pain and makes my code ugly. Or I could implement a separate cleanup mechanism for crashed processes. That’s a separation of concerns, and it’s more robust: signal handlers don’t help when the app is out of memory, a separate recovery does.

In relationships, vulnerability lets us form connections.[2] When emotional defenses are high, deep communication doesn’t happen. If we leave ourselves open to hurt, if we take that risk, there are payoffs. If we build a network of honest, compassionate friendships, those friends become our recovery mechanism for when we do get hurt. Friends who hug you and say “You are good. I love you.” Fall in love, get burned, cry to my people and be comforted.

In the software I currently work on, I take the strategy of building safety nets at the application, process, subsystem, and module levels, as feasible.[3] Then while I try to get my code right, I don’t convolute my code looking for hardware and network failures, bad data and every error I can conceive. There are always going to be errors I don’t conceive. Fail gracefully, and pick up the pieces. 

—–

[1] Someone tweeted a quote from some book on this, on the difference between weakness and vulnerability, a few weeks ago and it clicked with me. I can’t find the tweet or the quote anymore. Anyone recognize this?

[2] Brené Brown on Vulnerability. Twenty minutes that can change you or affirm you.

[3] The actor model (Akka in my case) helps with recovery. It implements “Have you restarted your computer?” at the small scale.

—–

I’m going to post a condensed version on my real blog.

Weakness and Vulnerability

Weakness and vulnerability are different. Separate the concerns: [1]

Vulnerability is an openness to being wounded.
Weakness is inability to live through wounds.

In D&D terms: vulnerability is a low armor class, weakness is low hit points. Armor class determines how hard it is for an enemy to hit you, and hit points determine how many hits you can take. So you have a choice: prevent hits, or endure more hits.

If you try to make your software perfect, so that it never experiences a failure, that’s a high armor class. That’s aiming for invulnerability.

Thing is, in D&D, no matter how high your armor class, if the enemy makes a perfect roll (a 20 on a d20, a twenty-sided die), that’s a critical hit and it strikes you. Even if your software is bug-free, hardware goes down or misbehaves.

If you’ve spent all your energy on armor class and little on hit points, that single hit can kill you.

Embrace failure by letting go of ideal invulnerability, and think about recovery instead. I could implement signal handlers, and maintain them, and this is a huge pain and makes my code ugly. Or I could implement a separate cleanup mechanism for crashed processes. That’s a separation of concerns, and it’s more robust: signal handlers don’t help when the app is out of memory, a separate recovery does.

In the software I currently work on, I take the strategy of building safety nets at the application, process, subsystem, and module levels, as feasible.[3] Then while I try to get my code right, I don’t convolute my code looking for hardware and network failures, bad data and every error I can conceive. There are always going to be errors I don’t conceive. Fail gracefully, and pick up the pieces.

—–
An expanded version of this post, adding the human element, is on True in Software, True in Life.

—–
[1] Someone tweeted a quote from some book on this, on the difference between weakness and vulnerability, a few weeks ago and it clicked with me. I can’t find the tweet or the quote anymore. Anyone recognize this?
[3] The actor model (Akka in my case) helps with recovery. It implements “Have you restarted your computer?” at the small scale.

Unreachable Stars

Something I love about agile and iterative development: acknowledgement that software is never done. It’s never the best it could be – even if it were, “best” changes quickly. A typical project is “done” by decision, not by any inherent completeness. We could always improve.

There’s a beauty in that, an embracing of imperfection and balance. And there’s a dichotomy of striving that defies traditional goals. Within any iteration and any one day, there’s a victory condition of completing a feature or small set of changes. Small victory conditions, achieved all the time. At the same time there’s a larger direction: a striving for software that delights people who use it. This is the Unreachable Star of an agile project, the target that is always in the distance, always moving away from us.

At the same time, within the team, there’s work toward becoming better as developers individually and collectively. Each retrospective has its victory conditions of the week, and we always find more ways to improve. The Unreachable Star is perfect communication, productivity, creativity. We aim for it, yet if we ever think we’ve achieved it, we stop growing and instantly fail.

Contrast with waterfall projects, which define “success” as a particular featureset by a particular date. Work really hard for a long time! Hit the milestone (if you’re really lucky), put up a banner, and then miss the real target. Software developed in isolation never delights people who use it. And then what? Athletes who work their whole lives and finally win gold – then what? Victory is fleeting, so keep it small and frequent. The big-milestone strategy lacks the vision of the Unreachable Star and the daily satisfaction of meeting victory conditions.

This works for me in life. Every day I have small victories, like writing this post, tweeting something that is meaningful to me, completing a task, hearing my daughter laugh. In the larger scale, I don’t have “goals” that I work toward blindly only because I set them for myself. Instead I have Unreachable Stars: as a developer, to keep learning higher levels of abstraction and to share it with others; as a partner, to be completely honest with my connections; as a mother, that my children know I love them and can grow in directions of their choosing.

There are many paths toward my Unreachable Stars, sometimes too many (property-based testing! scalaz-streams! clojure!). It isn’t important which I follow, nor is it important that I head straight there. Any small step in the general direction is a victory. If I define a milestone of some sort (“speak at conferences”) and then achieve that, it’s another victory for the day. If I don’t achieve it, there are many other paths to the Star. No milestone, achieved or missed, can end the quest.

Big milestones (what most people call “goals”) are valuable in defining a strategy toward the Star. They give us something to break down into our daily victory conditions. That’s all. They’re not the real objective, and they can distract us if we’re not careful. It is the mission, the quest, the target-that-is-only-aimed-for-never-reached, that inspires me.

I’ve been looking for a word for this, for the target that sets my direction with no expectation of hitting it. The song from Man of La Mancha, sung by Don Quixote, expresses it. Even in his delusion he has no expectation of fixing the world. Yet he finds happiness in every small move he can make toward rightness. Others are inspired by him, and start to move the world in their small ways. He spread hope, and so became larger than one person.

“And the world will be better for this, that one man, scorned and covered with scars, still strove with his last ounce of courage to reach the Unreachable Star." 

https://www.youtube.com/watch?v=-Mfd4E7kpvc

Testing in Akka: sneaky automatic restarts

Restarts are awesome when stuff fails and you want it to work. Akka does this by default for every actor, and that’s great in production. In testing, we’re looking for failure. We want to see it, not hide it. In testing, it’s sneaky of Akka to restart our whole actor hierarchy when it barfs.

Change this sneaky behavior in actor system configuration by altering the supervisor strategy of the guardian actor. The guardian is the parent of any actor created with system.actorOf(…), so its supervision strategy determines what happens when your whole hierarchy goes down.

Then, in your test’s assertions, make sure your actor hasn’t been terminated. If the guardian has a StoppingSupervisionStrategy and your actor goes down, it’ll stay down.

Code example in this gist.

Modularity in Scala: Isolation of dependencies

Today at work, I said, “I wish I could express the right level of encapsulation here.” Oh, but this is Scala! Many things are possible!

We have a class, an akka Actor, whose job is to keep an eye on things. Let’s pretend its job is to clean up every so often: sweep the corners, wash the dishes, and wipe the TV screen. At construction, it receives all the tools needed to do these things.

class CleanerUpper(broomBroom
                   rag: Dishcloth, 
                   wiper: MicrofiberTowel, 
                   config: CleanConfig) … {

  def work(…) {
    broom.sweep(config, corners)
    rag.wipe(config, dishes) 
    wiper.clear(tv)
  }
}

Today, we added reporting to the sweeping functionality. This made the sweeping part complicated enough to break out into its own class. At construction of the Sweeper, we provide everything that remains constant (from config) and the tools it needs (the broom). When it’s time to sweep, we pass in the parts that vary each time (the corners).[1]

class CleanerUpper(broom: Broom
                   rag: Dishcloth, 
                   wiper: MicrofiberTowel, 
                   config: CleanConfig) … {
  val sweeper = new Sweeper(configbroom)

  def work(…) {
    sweeper.sweep(corners)
    rag.wipe(config, dishes) 
    wiper.clear(tv)
  }
}

Looking at this, I don’t like that broom is still available everywhere in the CleanerUpper. With the refactor, all broom-related functionality belongs in the Sweeper. The Broom constructor parameter serves only to construct the dependency. Yet, nothing stops me (or someone else) from adding a call directly to broom anywhere in CleanerUpper. Can I change this?

One option for is to construct the Sweeper outside and pass it in, in place of the Broom. Then construction would look like

new CleanerUpper(new Sweeper(configbroom), rag, wiper, config)

I don’t like this because no one outside of CleanerUpper should have to know about the submodules that CleanerUpper uses. I want to keep this internal refactor from having so much impact on callers.

More importantly, I want to express “A Broom is needed to initialize dependencies of CleanerUpper. After that it is not available.”

The solution we picked separates construction of dependencies from the class’s functionality definition. I made the class abstract, with an uninitialized Sweeper field. The Broom is gone.

abstract class CleanerUpper
                   rag: Dishcloth, 
                   wiper: MicrofiberTowel, 
                   config: CleanConfig) … {
  val sweeper: Sweeper

  def work(…) {
    sweeper.sweep(corners)
    rag.wipe(config, dishes) 
    wiper.clear(tv)
  }
}

Construction happens in the companion object. Its apply method accepts the same arguments as the original constructor — the same objects a caller is required to provide. Here, a Sweeper is initialized.

object CleanerUpper {
  def apply(broom: Broom
            rag: Dishcloth,
            wiper: MicrofiberTowel, 
            config: CleanConfig): CleanerUpper = 
    new CleanerUpper(rag, wiper, config) {
      val sweeper = new Sweeper(config, broom)
    }
}

The only change to construction is use of the companion object instead of explicitly new-ing one up. Next time I make a similar refactor, it’ll require no changes to external construction.

val cleaner = CleanerUpper(broom, rag, wiper, config)

I like this solution because it makes the dependency on submodule Sweeper explicit in CleanerUpper. Also, construction of that dependency is explicit.

There are several other ways to accomplish encapsulation of the broom within the sweeper. Scala offers all kinds of ways to modularize and break apart the code — that’s one of the fascinating things about the language. Modularity and organization are two of the biggest challenges in programming, and Scala offers many paths for exploring these.

————-
[1] This example is silly. It is not my favorite kind of example, but all the realistic ones I came up with were higher-cognitive-load.