Monday, May 25, 2015

git: handy alias to find the repository root

To quickly move to the root of the current git repository, I set up this alias:

git config --global alias.home 'rev-parse --show-toplevel'

Now,  git home prints the full path to the root directory of the current project.
To go there, type (Mac/Linux only)

cd `git home`

Notice the backticks. They're not single quotes. This executes the command and then uses its output as the argument to cd.

This trick is particularly useful in Scala, where I have to get to the project root to run sbt compile. (Things that make me miss Clojure!)

Saturday, May 2, 2015

Fitting in v. Belonging

In your team, do you feel like you fit in? Do you have a feeling of belonging?

These are very different questions.[2] When I fit in, it's because everyone is sufficiently alike. We have inside jokes, TV shows or sports we talk about, opinions we share and common targets of ridicule. New people can fit in by adopting these opinions and following these sports.

When I belong, it's because everyone wants me to be there, because the group wouldn't be the same without me. We value each other for our differences. We have values that we share, and opinions we discuss. New people are integrated as we come to know and appreciate each other.

"Fitting in," that superficial team culture of common interests and inside jokes, is much easier to establish. And it's easier to hire for, because we can base choices on appearances and social cues. Hoodies and quotes from The Princess Bride. But it doesn't get us a strong team. On a strong team, people share ideas, they pull from their varied perspectives, they emphasize their differences because that's their unique contribution. This weaving together of various strengths, respect for the unexpected -- this is how a team can be stronger than its parts, this is where novel solutions come from. It emerges from feelings of belonging, which come from the group's deeper culture. That's much harder to establish.

On a wholehearted team, we show up as our whole selves and put all our creativity into the team's goals. How can we achieve this? Hire for value fit, not culture fit. Don't settle for the comfort of "fitting in" - aim for the safety of belonging.

I had this kind of team at my last job, at Outpace. We loved each other as people and respected each other as developers. And this was a remote team - we didn't fall back on physical proximity as appearance of teamwork. We shared goals, discussed them and evolved them. We shared our frustrations, both work and personal. When opinions clashed, we asked why, and learned. On this team, I explored ideas like the sea map. We grew individually and together.

That feeling of belonging makes it safe to express ideas and to run with them. And to take ideas from others and expand on them. Poof: innovation. Without that feeling of belonging, when the aim is to fit in, we express agreement with dominant voices.[1] Superficial cultural fit actively represses new ideas.

How can we move our teams toward a greater sense of belonging? Ask people about their interests that you don't share. Respect each person's experiences and opinions, especially when these are unique among the group. Instead of "We all agree, so we must be right," say, "We all agree. This is dangerous; can we find another view?" When pair programming, if you think your pair has the wrong idea, try it anyway. When someone says something dumb, their perspective differs; respond with curiosity, not judgement. Cherish our differences, not superficial similarities. Sacrifice the comfort of fitting in for the safety to be ourselves.

[1] Research has shown that teams of similar-looking people emphasize similarities. They're driven toward groupthink, quiet silencing of dissent. When someone breaks the uniformity, the not-obviously-different people starts expressing the parts of them that are unique. (I can't find the reference, anyone know it?)

[2] The dichotomy between fitting-in and belonging comes from BrenĂ© Brown's book, Daring Greatly.

Wednesday, April 29, 2015

Data v Awareness

In the computer industry, data and conscious thinking are praised, as opposed to an integrated awareness.[1] How is the work going? the task-tracking tools, the commits, and the build results provide data, but only conversations with the team can provide awareness. Awareness of mood and relationships and trends, of uncertainties and risks. Perhaps this is part of organizations' fear of remote work: colocation provides opportunities to read the mood of the team. Data alone can't provide that.

In sociological research like Brené Brown's, she starts with awareness: interviews, a person's story in context. Then she codes (in this context, "to code" is to categorize and label) the answers, and they become data. She aggregates that data to get a broader picture, and that leads to a broader awareness.

The key is: local awareness, to data, to aggregated data, to broader awareness.

On my last team, we were working on this. I wanted to track what was holding us back, and what was helping us move. Which tools in our technology stack cost us the most energy, and which improvements are paying off. To do this, we started posting in Slack whenever something frustrated us or helped us along, with custom emoticons as labels. For instance:
weight: clojure set operations behave unpredictably if passed a vector; lift: test-data generation utility for X service; weight: local elasticsearch version different from prod
This turns our awareness of the current situation into data, which a program can aggregate later. At retro time, I turned the words next to the hot-air balloon ("lift," because it helps us move the project up and forward) into a word cloud.[2] The words next to the kettlebell ("weight," because it's weighing down the balloon, holding us back) formed a separate word cloud. This gave us a visualization to trigger discussion.

The aggregation of the data produced a broader level of awareness in our retrospective. This contrasts with our remembered experience of the prior sprint. Our brains are lousy at aggregating these experiences; we remember the peak and the end. The most emotional moment, and the most recent feelings. The awareness -> data -> aggregation -> awareness translation gives us a less biased overview.

The honest recording of local awareness happens when the data is interpreted within the team, within the circle of trust, within context. There's no incentive to game the system, except where that is appropriate and deliberate. For instance, the week after the first word cloud, Tanya posted in the channel:
weight: elasticsearch elasticsearch elasticsearch elasticsearch elasticsearch
She's very deliberately inflating a word in the word cloud, corresponding to the level of pain she's experiencing. (context: we were using Elasticsearch poorly, totally nothing wrong with the tech, it was us.) Her knowledge of how the data would be used allowed her to translate her local awareness into a useful representation.

Data alone is in conflict with a broad, compassionate awareness of the human+technological interactions in the team. But if the data starts with awareness, and is aggregated and interpreted with context, it can help us overcome other limitations and biases of our human brains. In this way, we can use both data and awareness, and perhaps gain wisdom.

[1] "Computing: Yet Another Reality Construction," by Rodney Burstall, inside Software Development and Reality Construction
[2] Thank you @hibikir1 for suggesting the first reasonable use of a word cloud in my experience

The Quality Wheel

"Quality software." It means something different to everyone who hears it.

You know quality when you see it, right? Or maybe when you smell it. Like a good perfume. Perfume preferences are different for everyone, and quality means something different for every application.

In perfume, we can discover and describe our preferences using the Fragrance Wheel. This is a spectrum of scent categories, providing a vocabulary for describing each perfume, the attributes of a scent.
Floral notes (Floral, Soft Floral); Oriental notes (Floral Oriental, Soft Oriental, Woody Oriental); Woody notes (Mossy woods, dry woods); Fresh notes (citrus, green, water)
Perhaps a similar construction could help with software quality?

When a developer talks about quality, we often mean code consistency and readability, plus automated testing. A tester means lack of bugs. A designer means a great UI, a user means great experience and exactly the right features and lack of errors or waiting. An analyst means insightful reporting and the right integrations, a system administrator means low CPU usage and consistent uptime and informative logging. Our partners mean well-documented, discoverable APIs and testing tools.
Usability (Features, Discoverability, User Experience); Performance (Responsiveness, Availability, Scalability); Flexibility (Speed of Evolution, Configurability); Correctness (Visibilty, Automated Tests, Accuracy)
Each of these are attributes of quality. For any given software system and for each component, different quality attributes matter most. What's more, some aspects of quality compliment each other, each makes the other easier - for instance, a good design facilitates a great user experience. Readable code facilitates lack of bugs. Consistent uptime facilitates lack of waiting. Beautiful (consistent, modular, readable) code facilitates all the externally-visible aspects of quality.

However, other aspects of quality are in conflict. Quantity of features hurts code readability. More integrations leads to more error messages. Logging can increase response time.

If we add nuance to our vocabulary, we can discuss quality with more detail, less ambiguity. We can decide which attributes are essential to our software system, and to each piece of our system. Make the tradeoffs explicit, and allocate time and attention to carefully chosen quality attributes. This gets our system closer to something even greater: usefulness.

The quality wheel pictured above is oversimplified; it's designed to parallel the original version of the Fragrance Wheel. I have a lot more quality attributes in mind. I'd love to have definitions of each piece, along with Chinese-Zodiac-style "compatible with/poor match" analysis. If this concept seems useful to you, please contribute your opinions in the comments, and we can expand this together.

Sunday, April 26, 2015

Your Code as a Crime Scene: book review

What can we learn about our projects with a little data science and a lot of version control?
Locate the most dangerous code! Find where Conway's Law is working against us! Know who to talk to about a change you want to make! See whether our architectural principles are crumbling!

Adam Tornhill's book shows both how and why to answer each of these questions. The code archaeology possibilities are intriguing; he shows how to get raw numbers and how to turn them into interactive graphs with a few open-source scripts. He's pragmatic about the numbers, reminding the reader what not to use them for. For instance: "the value of code coverage is in the implicit code review when you look at uncovered lines." Trends are emphasized over making judgements about particular values.

Even better are Adam's expansive insights into psychology, architecture, and the consequences of our decisions as software engineers. For instance: we know about the virtues of automated tests, but what about the costs? And, what is beauty in code? (answer: lack of surprise)

There's plenty of great material in here, especially for a developer joining an existing team or open-source project, looking to get their mind around the important bits of the source quickly. I also recommend this book for junior- to mid-level developers who want to learn new insight into both their team's code and coding in general. If you want to accelerate your team, to contribute in ways beyond your own share of the coding, then run Adam's analyses against your codebase.

One word of caution: it gets repetitive in the intro and conclusion to the book as a whole and each section and each chapter. Whoever keeps repeating "Tell them what you're going to tell them, tell them, tell them what you just told them," can we please get past that now??

A few factoids I learned today from this book:
- Distributed teams have the most positive commit messages
- Brainstorming is more effective over chat than in an in-person meeting

and when it comes to the costs of coordination among too-large teams: "The only winning strategy is not to scale." (hence, many -independent- teams)

Post-agile: microservices and heads-up development

Notes from Craft Conference 2015, Budapest.

Craft conference was all about microservices this year.[1] Yet, it was about so much more at the same time -- even when it was talking about microservices.

lobby of the venue. Very cool, and always packed
Dan and I went on about microservices in our opening keynote,[2] about how it's not about size, it's about each service being a responsible adult and taking care of its own data and dependencies. And being about one bounded context, so that it has fewer conflicting cross-cutting concerns (security, durability, resilience, availability, etc) to deal with at any one time.

But it was Mary Poppendieck, in her Friday morning keynote,[3] who showed us why microservices aren't going away, not any more than the internet is going away. This is how systems grow: through federation and wide participation. (I wish "federated system" wasn't taken by some 1990s architecture; I like it better than "microservices.") Our job is no longer to control everything all the computers do, to make it perfectly predictable.[a]

Instead, we need to adapt to the sociotechnical system around us and our code. No one person in can understand all the consequences of their decision, according to Michael Nygard.[4] We can't SMASH our will upon a complex system, Mary says, but we can poke-poke-poke it; see how it responds; and adjust it to our purposes.

What fun is this?? I went into programming because physics became unsatisfying once I hit quantum mechanics, and I couldn't know everything all at once anymore. Now I'm fascinated by systems; to work with a system is to be part of something bigger than me, bigger than my own mental model. This is going to be a tough transition for many programmers, though. We spent our training time learning to control computers, and now we are exhorted to give up control, to experiment instead.

And worse: as developers must adapt, so must our businesses. In the closing keynote,[5] Marty Cagan made it very clear that our current model is broken. When most ideas come from executives, implemented according to the roadmap, it doesn't matter how efficient our agile teams are: we're wasting our time. Most ideas fail to make money. And the ones that do make money usually take far longer than expected. He ridicules the business case: "How much revenue will it generate? How much will it cost?" We don't know, and we don't know! Instead of measuring the impact of an idea after months of development, product teams need to measure in hours or days. And instead of a few ideas from upper management, we need to try out many ideas from the most fruitful source: developers. Because we're most in the position to see what has just become possible.

Exterior of the venue! (after the tent is down.)
I'd say "developers are a great source of innovation," except Alf Rehn reports that the word has been drained of meaning.[6] Marty Cagan corroborates that by using "ideas" throughout his keynote instead of "innovation." So where do these ideas come from? Diversity, says Pieter Hintjens,[7] let people try lots of things. Discovery, says Mike Nygard, let them see what other teams are doing.

Ideas come from having our heads up, not buried only in the code. They come from the first objective of software architecture: understanding the business problem. They come from handing teams an objective, NOT a roadmap. Marty Cagan made that point very clear. Adrian Trenaman concurred,[8] describing how Gilt teams went from a single IT to a team per line of business to a team per initiative. It is about results, measured outcomes.

All these measurements, of results, of expectations, of production service activity, come down to my favorite question - "How do we know what we know?"[b] Property-based (aka generative) testing is experiencing a resurgence (maybe its first major surgence) lately, as black-box testing around service-level components. In my solo talk,[10] I proposed a possible design for lowering the risk around interacting components. Mary had some other ideas in her talk too, which I will check out. Considering properties of a service can help us find the seams that align simplicity with options.

Mike Nygard remarked that the most successful microservices implementations he's seen started as a monolith, where refactoring identified those seams. There's nothing wrong with a monolith when that supports the business objectives; Randy Shoup said that microservices solve scaling problems, not business problems.[9] Mike and Adrian both pointed out that a target architecture is not a revolution, but an evolving direction. Architecture is like a city: as we build microservices in the new, hip part of town, those legacy tenements are still useful. The architecture is done only when the company goes out of business. Instead of working to a central plan, we want to develop situational awareness ("knowing what's happening in time to do something about it"[3]), and choose to work on what's most important right now.

It isn't enough to be good at coding anymore. The new "full-stack" is from network to customer. Marty: if your developers are only coding, you're not getting half their value. I want to do heads-up development. "Software Craftsmanship is less about internal efficiency, and more about engaging with the world around us," says Alf Rehn. "Creators need an immediate connection to what they are creating," quotes Mary Poppendieck.

As fun as it is to pop the next story off the roadmap and sit down and code it, we can have more impact. We can look up, as developers, as organizations. We can look at results, not requirements. We can learn from consequences, as well as conferences.

This transition won't be easy. It's the next step after agile. Microservices are a symptom of this kind of focus, the way good retrospectives are associated with constant improvement. Sure, it's all about microservices - in that microservices are about reducing friction and lowering risk. The faster we can learn, the farther we can get.

I'll add the links as Gergely posts the videos.

[1] Maciej was starting to get bored
[2] my keynote with Dan, "Complexity is Outside the Code"
[3] Mary Poppendieck's keynote, "The New New Software Development Game"
[a] Viktor Klang: "Writing software that is completely deterministic is nonsense because no machine is completely deterministic," much less the network.
[4] Mike Nygard's talk, "Architecture Without an End State"
[5] Marty's keynote
[6] Alf Rehn (ah!  what a beautiful speaker! such rhythm!) keynote. Maybe he didn't allow recording?
[7] Pieter's talk
[8] Adrian's talk, "Scaling Micro-services at Gilt"
[b] OK my real favorite question is "What is your favorite color?" but this is a deep second.
[9] Randy's talk, "From the Monolith to Microservices"
[10] my talk, "Contracts in Clojure: a compromise between types and tests"

Charting team course with a Seamap

Great teams are focused on results, not on projects. But how many of us really see the results of our efforts? Can we track the impact of what we're doing today all the way to its hoped-for impact on customers or users?

A project roadmap doesn't cut it. It doesn't give us a compelling picture of what we're accomplishing. It acts like there is only one path toward our goals. A roadmap fences us in, reduces options, squashes ideas.

We want a visualization that lets us see the business goals, from the customer to the organization to the team to the individual or pair. From ages to iterations to the current day. We want to document our aims at each level, and record our attempts and the problems we faced. We don't want to restrict ourselves to the first plan someone came up with. We want to reach the right milestones, discard others, learn -- all without losing perspective of the larger goals.[2]

Marty Cagan said he knows roadmaps are terrible, but he didn't suggest an alternative. I have one: it's called the Seamap.

A seamap represents our journey as a mountain across an ocean. We won't sail straight for it across the deep water; we'll stop at many islands along the way. This is iterative delivery. We don't have to travel in a straight line, because we're exploring the solution along the way. Islands we thought we would visit may become less interesting, while new ideas erupt and solidify.

At each milestone in the seamap, we can zoom in. That part of the ocean contains the likely stops on that piece of the journey. Each day, as we discover tasks, we can add that. Achievements can be recorded with little flags. Surprises are represented by a sea monster; we might defeat it, or route around it by accomplishing our day's objective some other way.

The zoom can go as deep as necessary; milestones we thought were close often are more treacherous than expected. That's why I use Prezi to create the seamaps: every level of scale is explored in one document.

At the very top, above the horizon, above the highest-level goal, are the Unreachable Stars. These are guiding principles that direct our journey, even though they're never perfectly achieved. These are the aspects of quality that matter for that project, or ethics that matter to the organization. My projects all include "evidence this will work," which encompasses market research, proof of correctness, and automated testing. We never have perfect testing, but any task that takes us closer to that star is valuable in more ways than its immediate progress toward a milestone.

On a daily basis, I use the seamap in standup. As each developer or pair states their plans for the day, I place their pictures (trello icons) on the place on the map. We can see whether we're all spread out, or working closely. Other images mark points of interest: drop monsters into the picture for problems, a sea arch if we plan a nontrivial deploy.[4]

On an iteration basis, as well as anytime I ask myself whether I'm working on the most important thing, I zoom out in the seamap.[1] Then I can ask, is this still the best route to our ultimate goal?

The seamap provides perspective, planning, and task organization. It does not pretend that we already know the single route to our destination. It can show what we have done, where we are now, and where we are going. And, it's beautiful to look at![3]

Try this on your team: copy this mostly-blank template, or look at other samples on my Prezi profile. If your team is colocated, draw this stuff on a white board or a poster. Draw on post-its, print images and tape them, then move them around the next day. A physical seamap is the best way to find out what works for you. I started with a giant post-it board on my bedroom wall, and held it up in front of the camera for remote stand-ups.

Marty Cagan also said: if your developers are only for code, you're getting only half their value. We aren't typists; we're thinkers. Support and encourage thinking, ideas, innovations by sailing off the roadmap into a world of wider options.

Feedback and suggestions welcome!

[1] Mary Poppendieck says that in the military, one must understand the intent of orders two organizational levels up, and maintain situational awareness ("when something goes wrong, we'll know in time to do something about it") one level up.

[2] Drawing a line and sailing straight along it may seem efficient, until the ship sinks in the middle of nowhere. Instead, stop at a port each night (deploy some code), and we have a solid place to come back to.

[3] Somewhere there's going to be a spreadsheet, or Trello, or Target Process, or Jira, or something containing the nitty-gritty of each milestone and task. Don't let the linear, one-dimensional format of those limit you. Label or link the seamap images, refer to those more detailed descriptions as needed. Keep upper management happy enough while your team explores and learns.

[4] I use other maps to show tasks like firefighting and hardening of the current system. These are based around data flow in the existing system, typically. That's another blog post.