These are not the only options. Wineglass edition

Today I found myself in the kitchen, near the fridge with the wine (it’s an excellent Chardonnay from Sonoma, thanks XYZ Beverages in Tybee Island, you exceed my expectations although you don’t have a website to link to). My empty glass was out on the screened porch.

Do I go outside for the glass? Or take the wine bottle to the glass, and then return it to the fridge?

These are not the only options. I snag another wineglass from the cupboard, fill it with wine, and take that out to the porch.

Now I have two dirty wineglasses, but who cares? The dishwasher washes them all at the same rate.

This is garbage collection in action. The dishwasher acts as a garbage collector for dirty dishes. It adds the capability of “do not worry about how many dishes you dirty. They will all be cleaned for the same fixed cost that you have already incurred.”

This removes one consideration that I need to think about in my actions. I’m free to optimize for my higher-level objectives (“be on the porch, with wine in a glass”) while ignoring the accumulation of resources (dirty wineglasses). It takes some adjustment to benefit from this condition.

It takes some adjustment in a brain to move from scarcity (“Dishes are a resource with a cost”) to abundance (“dirty dishes meh, not a problem anymore”). Once adjusted, the options opened to me are widened, in a way that a clearly optimal path is opened.

Now pardon me while I finish this delicious glass of wine and fetch another, from the nice cold bottle still in the fridge.

Zooming in and out, for software and love

The most mentally straining part of programming (for me) is focusing down on the detail of a line of code while maintaining perspective on why we are doing this at all. When a particular implementation gets hard, should I keep going? back up a step and redesign? or back way up and solve the problem in a different way?

Understanding the full “why” of what I’m doing helps me make decisions from naming to error handling to library and tool integrations. But it’s hard. It takes time to shift my brain from the detail level to the business level and back. (This is one way pairing helps.)

That zooming in and out is tough, and it’s essential. This morning I learned that it is also essential in love. Maria Popova quotes poets and philosophers on how love requires understanding, then:

We might feel that such an understanding calls for crouching closer and closer to its subject, be it self or other, in order to examine it with narrow focus and shallow depth of field, but this is a misleading intuition — the understanding of love is an expansive understanding, requiring us to zoom out of our habitual solipsism so as to regard ourselves and the object of our love from a great distance against the backdrop of universal life.

Maria Popova, Brain Pickings

Abeba Birhane, cognitive scientist, points out that Western culture is great at picking things apart, breaking problems up to their smallest possible components. She quotes Prigogine and Stenges: “We are so good at it. So good, we often forget to put the pieces back together again.”

She also sees this problem in software. “We forgot why we are doing it, What does this little component have to do with the big picture?” (video)

Software, love, everywhere. Juarrero brings this together when she promotes hermeneutics as the way to understand complex systems. Hermeneutics means interpretation, finding meaning, especially of language. (Canonical example: Jews studying the torah, every word in excruciating detail, in the context of the person who wrote it, in the context of their place and time and relations.) Hermeneutics emphasizes zooming in and out from the specific words to the work as a whole, and then back. We can’t understand the details outside the broader purpose, and the purpose is revealed by all the details.

This is the approach that can get us good software. (Not just clean code, actual good software.) I recommend this 4-minute definition of hermeneutics; it’s super dense and taught me some things this morning. Who knows, it might help your love life too.

Certainty, Uncertainty, or the worst of both

Des Cartes looked for certainty because he wanted good grounds for knowledge, a place of fixity to build on, to make predictions.

Juarrero counters that uncertainty allows for novelty and individuation.

In software, we like to aim for certainty. Correctness. Except in machine learning or AI; we don’t ask or expect our algorithms to be “correct,” just useful.

The predictions made by algorithms reproduce the interpretations of the past. When we use these to make decisions, we are reinforcing those interpretations. Black people are more likely to be arrested. Women are less likely to be hired.

Machine learning based on the past, choosing the future — this reinforces bias. It suppresses novelty and individuation. It is the worst of both worlds!

This doesn’t mean we should eschew this technology. It means we should add to it. To combine the fluidity of the human world with the discreteness of machines, as Kevlin Henney puts it. We need humans working in symmathesy with the software, researching the factors that influence its decision and consciously altering them. We can tweak the algorithms toward the future we want, beyond the past they have observed.

Machine learning models come from empirical data. Logical deduction comes from theory. As Gregory Bateson insisted: progress happens in the interaction between the two. It takes a person to tack back and forth.

We can benefit from the reasoning ability we wanted from certainty, and still support novelty and individuation. It takes a symmathesy.

This post is based on Abeba Birhane’s talk at NCrafts this year. Video

For cleaner code, write ugly code

We want to write only clean code, right?

Wrong. I want to write eventually-clean code. It starts exploring a space, and then I refine it to be cleaner and more suited to purpose. Usually, that purpose becomes clearer through writing, reading, and using the code.

That process of refining or tidying up can feel tedious, compared to implementing more features. It can be tempting to leave off error handling. I have a strategy for that: meaningful ugliness.

When I’m prototyping, I make the code so ugly that it will be satisfying to clean up. No attempt at appearing clean. I put a bunch of casts, bad casing maybe, random names instead of plausible-but-inaccurate ones. No null-checking.

for (const gerald of allKindsOfStuff.fingerprints) {
    (gerald as any).displayName =
         allKindsOfStuff.feature.convertinate(gerald.name);            }

(exaggerated to show detail)

When cleaning, I often start by making the code uglier. To move from an OO style toward functional, start by replacing all global or class-level variables with parameters. Twelve parameters on a function?! That’s hideous! Yes. Yes. Let ogres look like ogres.

This lets me feel productive when I come back to the code and clean it up. Later, I know more about what this code is for, what might be null and what won’t, what dependencies I can eliminate and which are meaningful. This is a better time to clean.

Disguising rough code in socially-acceptable clothing prevents cleaning. Appearance-of-good is the enemy of better.

The next architecture book you must read

Today, another tweet about “how can I write the cleanest, best architected code?” gets piles of book references in response.

Yes, we want to be good at writing code. We want to write the best code. The best code for what? “Writing code” is an abstraction, like a transitive verb without an object. I can’t just “write code,” I must “write code to….”

The work is software development is not typing, it is making decisions. To make those decisions, we have to understand the details of code and technology, yes. We also have to understand the context and purpose, what we are writing the code to do.

My advice for “What should I read in order to write better code?” is usually, a book or magazine or internal memos about the business. Better is having conversations about the business with the experts inside your company, and to do that well, you need the vocabulary.

We need both the specific technical understanding and the business understanding. It’s so much easier to push for technical understanding. Because the business understanding is specific to each context. I can’t make a wide-audience tweet recommending a book, you have to find that closer-in.

Supplement Twitter with kitchen conversations or internal Slack channels that give you a broader perspective on the purpose of your work in the specific context you work in.

Zooming in and zooming out

When the db is the interface

There are two huge sources of inertia in software: data, and interfaces.

When two systems connect to the same database, you combine both. Ow.

When some other system is doing reporting on my database, I can’t change my table structure. That severely limits any internal reorganizations I might do. Especially when I don’t know how they use it, so I am afraid to touch it.

Then what if it’s a schemaless database? This means the schema resides in the application. When the schema resides in two applications at once, designing change is an exercise in hope.

Sometimes two systems access the same database because the database is the interface. We have an example of that at Atomist currently: one system (called squirrel, I don’t know why) populates a database, and another system (I call it org-viz) uses that data to make visualizations. How can that be okay?

Database as interface is not horrifying when:

  1. The database is not the system of record. We could repopulate all or part of it from elsewhere, so it’s somewhat disposable.
  2. The database has a schema. Interfaces must be well-defined.
  3. One system updates, the other reads.

In our case, we use PostgreSQL, so we have a schema. Some of our fields are JSON, which gives us extensibility, while the essential fields are guaranteed. The populating system (squirrel) operates from a queue, and we can replay the queue as needed, or play it into multiple databases. There are options for designing change.

Database as an interface is never going to be the cleanest decoupling, but it is not unreasonable when it is carefully designed and the teams don’t mind talking to each other. When the database is accidentally an interface, then you’re horked.

Inertia in the interface

What makes software hard to change?

As a developer, it’s easy to focus on the internal properties of the software system. The code needs refactored, the framework is old, we need more tests, or else fewer tests.

If your software is in production, these are not the biggest obstacle to change.

The important changes are the ones visible to the outside world, the ones that change the interface. And changing the interface means more than your software has to change: the systems that use your software need to change.

The more useful your software is, the more other systems depend on it, the scarier change is. You’re risking more than your piece of the world. You need progressive delivery, careful data migrations. Backwards compatibility: twice as many tests, and special cases in the code. You need to design the whole change.

And then, for your work to be useful, people have to use it. You must get the word out through social channels: talking to people, advocacy, (dare I say it) marketing.

This is all fine. It is right. No complaints.

Because our work is not to change software, it is to change a system. To add and broaden capabilities in systems larger than ours.

If our code is full of if/else statements for backwards compatibility, if our database integration is cluttered by a migration, if we spend more time crafting the deploy strategy than we did writing the code, cool.

The more people use the code, the harder it is to change, and the more valuable.

Caveat: yes, there are performance and scale improvements that are useful. A wider variety of functionality changes are useful, and these are more frequent, so that’s what I’m talking about here.

For more on this topic, see: From Puzzles to Products

Soft, or hard like mud

Soft skills are hard. “They take work to build and work to apply.”

@ruthmalan

The word “hard” describes sciences like physics and chemistry. It is confusing that “hard” can mean difficult, because these sciences aren’t more difficult than the “soft” ones like sociology and anthropology. They’re differently difficult.

The “hard” sciences are hard because they’re solid. We can stand on them. Physical laws are universal laws. They demand rigor, and rigor means proving that assertions apply in all cases, both through deduction and by checking against evidence.

The “soft” sciences are differently hard. They study humans systems, far too complex to make universal causal predictions. Conclusions about one culture or group are valid in some other groups and invalid in others. Rigor in complexity means studying which cases your assertions apply to. It means observing, discerning, and wallowing in context.

I describe my career progression from solving puzzles to growing products like this:

Correctness, puzzles, and the “hard” sciences are hard like rocks. Rock climbing is very technical. It takes a lot of skill and strength and hard work to climb. At the top of the cliff, you know you’ve achieved something specific.

Change, people, and the “soft” sciences are hard like mud. Like wading through goopy mud. Techniques can help, but each depends on the kind of mud. Strength helps, but pushing too hard can get you more stuck. It always helps to be okay with getting messy. When you finally reach that piece of relatively solid ground, the view is still a swamp.

Why would you want to wade through mud?

why does a fish want to swim through water?

People, interrelationships, change — this is our world. Sometimes we can carve out puzzles we can solve for real. The few universal laws we can find are priceless. But the rest of it — life is in the mud, in the deep context. The skills to navigate it are not easy. They are not satisfying in the same way, either. But they are how we find meaning, how we participate in a system bigger than our own self.

I am okay with getting messy.

Knowledge resides in teams

The magic of a gelled team is that they know how to work together, and together, they know how to do particular work. The members don’t know how to work together; the team does.

These learnings don’t reside in the members individually, the learnings are in the interrelations.

A shared know-how is jointly constructed between the participants. This shared know-how does not amount to the sum of the individuals’ know-hows nor does it strictly “belong” to any of the participants…. It involves instead the practice of coordinating sensorimotor schemes together, navigating breakdowns, and it belongs to the system the participants bring forth together: the dyad, the group, the family, the community, and so on.

“Linguistic Bodies The Continuity between Life and Language” Ezequiel A. Di
Paolo, Elena Clare Cuffari, and Hanne De Jaegher, quoted by @theblub

If you’re a director who gets to decide which teams stay together and which break apart, you have a lot of power — and very little control. Power can bust up symmathesies, but not build them or repair them. Other levels of hierarchy can set up conditions for success, but teams grow from within.

Leaving a company is scary because we know how to be in that company. Our own knowing-how-to-be exists partly in our interrelations there. Finding a new job means discovering a new way, a new self, to be in the new place. With families, even more so – this is part of what makes divorce so scary. If you’re in an unhealthy system and can’t imagine anything else, this is normal.

When you do get to be part of a healthy team, or a healthy family, appreciate it. Cherish it and nourish it.

Feature interaction

Why is legacy software so much harder to work in?

Why does development velocity slow down as a system grows?

Lots of reasons, sure, but I suspect the biggest one is oft unspoken.

Every new feature comes with the invisible requirement: “and everything else still works.”

Every existing feature, and even past bugs, makes every new feature harder. Every user with expectations is a drag on change.

Feature interactions are hard! And they surprise you. My favorite examples come from The Sims release notes:

  • Fish are no longer duplicated in the fridge when moving homes.
  • Televisions no longer play video after they are burned or broken.
  • Sims will no longer walk on water to view paintings placed on swimming pool walls.
  • Pianists will no longer continue playing pianos that have been detonated.
  • Sims will no longer receive a wish to “Skinny Dip” with Mummies.
  • Sims who are on fire will no longer be forced to attend graduation before they can put themselves out.
  • Sims can no longer “Try for Baby” with the Grim Reaper.

Your feature interactions are probably not this much fun.

Software gets harder to change as it becomes part of a larger system, with more users and more uses. Development slows down because it’s useful, and we want it to stay useful. This also makes each change more valuable, because it’s helping more people in more ways.

Don’t be discouraged. Do be diligent, and be okay with being slower. This is the price of success.