How can we develop and operate increasingly useful software?

Most software gets harder to change as it ages. Making modern applications, it is not enough to write a system and put it out there. We need continual improvement and adaptation to the growing world.

How can we develop and operate increasingly useful software?

To answer this, I need a mental model of how software teams work.

My model is symmathesy: a learning system of learning parts. A great software team is made of people, software, and tools. The people are always learning, the software is changing, the tools are improving.

With this view, let’s look at the question.

How can we (the people, tools, & software in a symmathesy) develop & operate (for learning, these must be together) increasingly (the ongoing quest) useful (the purpose; to learn, we need connection to its use) software (output of our system to the wider world)? People are products of our interactions; the future matters, so we need healthy growth; and code is a product of our interactions with it — so the route we take, or the process, matters.

The external output of the team is: useful software, running in production. To make it increasingly useful, grow the symmathesy. Growth in a system is defined as an increase in flow. In a symmathesy, that means growth is an increase in learning.

The people learn about the software by operating it, mediated by tools. Then we teach the software and tools by developing. Unless development & operations are in the same team, learning is blocked.

To increase usefulness, the team needs to learn about use. We can get input from outside, and study events emitted by the running software, mediated by tools.

What we know about usefulness leads the team to design change, and move the system to the next version of itself. That next version may be more useful to the outside, or it may be better at learning and teaching, the better to increase knowledge inside the team.

Someone today told me, “All great things in life come from compounding interest” — from feedback loops. (Compound interest is the simplest familiar example of a feedback loop.)

In a great software team, that “compound interest” is our relevant knowledge. Of the software, of its environment, of the people and systems who use it.

Maximize learning to increase the usefulness of software, at an accelerating pace.

Do things right in VSCode

Here are some settings that I change when I install VS Code on a new computer.

Change these by searching for them in the settings UI (Ctrl-comma on Windows or ⌘-, on Mac). Or open the JSON and paste them in; access the JSON from the command palette (Ctrl-Shift-P) by searching for Preferences: Open Settings (JSON).

autosave

Please save my files.

"files.autoSave": "onFocusChange",

autoformat

Please format my files. This only works on files associated with an installed formatter extension.

"editor.formatOnSave": true,

goodbye minimap

My screen is too small to want this miniaturized representation of my file at the right.

"editor.minimap.enabled": false,

goodbye activity bar

Those buttons on the left, I can get to them from the menu. Give me real estate.

"workbench.activityBar.visible": false,

Another thing to know, especially when you have the activity bar off: the Explorer window (or Debug, or Extensions, or whichever you have open) toggles with Ctrl-B (or ⌘-B 0n Mac).

Explorer window opens with Ctrl-Shift-E (⌘-shift-e on a Mac).

window title

I like the title of my window to be only the filename. And I think this gives some indication of whether it needs saved.

"window.title": "${dirty}${rootName}",

open projects in their own window

At a command line (outside VSCode), when I type code . please open a new VSCode window. Do not add this folder to an existing, open project.

"window.openFoldersInNewWindow": "on",

Also, make it big.

"window.newWindowDimensions": "maximized",

stop it with popping stuff up all the time

Especially when live coding, I don’t want stuff popping up on my cursor all day. It bothers me.

 "editor.quickSuggestions": {
        "other": false,
        "comments": false,
        "strings": false,
    },

no carriage returns please

On Windows, when creating a new file, please don’t use carriage returns in the line endings. We don’t need those anymore.

"files.eol": "\n",

all together now

Here are all of them in one cut-and-paste-able block for my next settings.json.

 { 
    "files.autoSave": "onFocusChange",
    "editor.formatOnSave": true,
    "workbench.activityBar.visible": false,
    "editor.quickSuggestions": {
        "other": false,
        "comments": false,
        "strings": false,
    },
    "window.openFoldersInNewWindow": "on",
    "editor.minimap.enabled": false,
    "window.title": "${dirty}${rootName}",
    "window.newWindowDimensions": "maximized",
    "files.eol": "\n",
}

Basements and galleries

In a big new project, where do you start?

My dad bought this old building, and he’s turning it into an antique mall. It’s a huge project: two floors of shopping plus a gallery and museum at the top. Yesterday he showed me what he’s done so far.

In the basement, he added support beams and repaired the sprinkler system. They painted all the water pipes red, air blue, and electric gray. Paint is stored where it won’t freeze. The elevator machinery is oiled and accessible.

On the third floor, where art and artifacts will draw people through the store, he has crafted walls of sunrise, sunset, and nightfall from wood siding in dozens of different stains.

He started with infrastructure and with beauty. The underpinnings of the whole store, made observable with color coded pipes. And the unique features that will make this store different from any other.

In a big project, there’s not one place to start; usually we need to make little circles. Foundations, surface, various points in between.

Emphasize the parts that make the work smoother: a clean workspace, visibility into what’s happening, fire prevention. And the parts that make the work fulfilling: the beauty, so we remember why we’re doing this project at all.

Rebase on the World: personal shell choice

“Why use bash when you have PowerShell?” <– words I did not expect to hear from my own mouth.

Over the past few weeks I’ve begun learning PowerShell, and it’s an improvement over the UNIX (and family) shells, bash and ksh etc.

PowerShell is newer. It builds on what those shells did right, and then gets it more right.

The UNIX-y shell commands have this magical power of composition: you can pipe the output of one to the input of another, and chain these to build tiny programs right on your command line. One sends text to STDOUT, the next reads it on STDIN.

PowerShell does this piping, except with objects instead of lines of text. The data is structured and interrogable (I can ask it what fields it has). In bash, you output your data as text, and the next program parses that text.

Here’s an example from Chapter 1 of PowerShell in Action. In bash, the sort command can parse the piped text and sort on it. You can specify numeric sorting.

ls -l | sort -k 5 -r -n

In PowerShell, the sort command works on named properties of the piped object. And it knows the type of the property, so nobody has to tell it what’s a number.

ls | sort -Property length -Descending

More: PowerShell standardizes parsing of command-line arguments. This gives me consistency when I use the command, and saves painful work when I write a command.

More: PowerShell gives me multiple output paths, one for data (to the next program) and another to the user who typed the command. In bash, commands like git abuse STDERR to send info back to the user.

When PowerShell was only for Windows, by far the most powerful command-line shell for me was bash (or fish or zsh, pick your favorite). Bash worked in the worlds I lived in, and that matters most. Now that PowerShell runs on Mac and Linux, it is the most powerful command-line shell for me … except for one thing.

The biggest thing bash has on PowerShell is: I already know bash. I can type stuff there and it just works. With PowerShell I have to look things up all the time.

But every time I look something up, I increase my abilities. PowerShell raises the floor I build on, compared to bash.

It is always a tradeoff between sharpening the tools we have, vs trading them in for a better model. In this case, I’ll take some slowness in the beginning for a faster top speed.

By switching, I gain advantages that the software world has created since I started programming twenty years ago. I rebase myself on the latest version of the world.

Increase capacity, move slower

If the highways are crowded, and they build more lanes, the highways get more crowded.

If development is slow, and you add resources, development gets slower.

Adding people to a project increases the capacity for activity. Activity doesn’t translate to outcomes.

In these cases, you’re adding capacity to the system for cars, or for work, but those aren’t what makes the system run faster. Instead, adding capacity for traffic or for activity leads the system to change in ways that generate more traffic or activity. Which gets in the way of flow.

(lots more examples in this article)

What you want instead is to make flow easier. Add trains, intersperse commerce and residential. Add continuous delivery, add support structures to make progress easier. Don’t add more capacity for work! Doing work isn’t the point! Make the path shorter, instead.

Game science, programming science

In Pokémon Go last summer, a new feature popped up: bad guys. Some Pokéstops turned dark, and Team Rocket appears there, and they want to fight me. And they always win, dammit!

the Team Rocket Grunt gloats over his victory.

This much I can discover within the game. If I want to know more — like how to win the fights — I turn to the internet.

People on the internet have cataloged all the Pokémon. Each has a type, and each type is weak to attacks of certain other types. They’ve also noticed that when the Team Rocket Grunt boasts, they reveal something about the type of Pokémon they have. After they boast, I get to choose which of my Pokémon to fight their Pokémon with.

People observed these properties of the in-game world, and they reasoned about which Pokémon will be effective against which Team Rocket Grunts, and they’ve tested these guesses in-game, and they’ve posted the results online. I can use their analysis to increase my effectiveness in the world of Pokémon Go.

People take it even farther: they’ve deduced the hidden talent levels of individual Pokémon based on observable properties, and created calculators so that you can evaluate your Pokémon. There is systematic testing and measurement here, and logical deduction, and math.

This is science. People do science to describe a Nature that exists inside this game world. And then they publish their results.

It’s a different culture from the Science we learn in school, that institution/body of knowledge/discipline/people that studies Nature in the physical world. But it is science: it is people using observation, causal hypotheses, tests, and discussion to increase a shared body of knowledge about how a world works.

In software development, StackOverflow works this way too. Questions and answers on StackOverflow are scientific publications, sharing observations and knowledge and recommendations about a Nature that exists inside a particular programming language, tool, or library.

Every piece of software constructs its own little reality. Collaborating online, we study them together. We do science.

Work with the business, not for it

Scientists should be on tap but not on top.

Winston Churchill

In the Cold War, political and technical considerations were no longer separable. The President got a Science Advisory Committee, but “apparently… scientists must not concern themselves with devising and proposing policies; they ought to limit themselves to answering such technical questions as they may be asked.” (Leo Szilard, physicist)

Yikes! Sounds like old style software development, where the programmers receive the requirements from the business.

I think we’ve learned better than that. Many of the most successful companies are led by technical people. We need the business experts and software developers working together. The business doesn’t know all the questions developers can answer, and devs don’t know what questions to ask the business — until we start implementing. Then, necessary questions rise to the surface, and lead to discussions which include more useful questions.

If developers are “on tap” as a resource, we can’t create anything better than you can specify (and believe me, that list of requirements is no specification). Our collective imagination is better than either alone.

Asking useful questions is the hard part. Collaborate on it.

Line endings in git

Git tries to help translate line endings between operating systems with different standards. This gets sooo frustrating. Here’s what I always want:

On Windows:

git config --global core.autocrlf input
This says, “If I commit a file with the wrong line endings, fix it before other people notice.” Otherwise, leave it alone.

On Linux, Mac, etc:

git config --global core.autocrlf false
This says, “Don’t screw with the line endings.”

Nowhere:

git config --global core.autocrlf true
This says, “Screw with the line endings. Make them all include carriage return on my filesystem, but not have carriage return when I push to the shared repository.” This is not necessary.

Windows and Linux on the same files:

This happens when you’re running Linux in a docker container and mounting files that are stored on Windows. Generally, stick with the Windows strategy of core.autocrlf=input, unless you have .bat or .cmd (Windows executables) in your repository.

The VS Code docs have tips for this case. They suggest setting up the repository with a .gitattributes file that says “mostly use LF as line endings, but .bat and .cmd files need CR+LF”:

* text=auto eol=lf
*.{cmd,[cC][mM][dD]} text eol=crlf
*.{bat,[bB][aA][tT]} text eol=crlf

Troubleshooting

When git is surprising you:

Check for overrides

Within a repository, the .gitattributes file can override the autocrlf behavior for all files or sets of files. Watch out for the text and eol attributes. It is incredibly complicated.

Check your settings

To find out which one is in effect for new clones:
git config --global --get core.autocrlf

Or in one repository of interest:
git config --local --get core.autocrlf

Why is it set that way? Find out:
git config --list --show-origin
This shows all the places the settings are set. Including duplicates — it’s OK for there to be multiple entries for one setting.

Why does this even exist?

Historical reasons, of course! (If you have a Ruby Tapas subscription, there’s a great little history lesson on this.)

Back in the day, many Windows programs expected files to have line endings marked with CR+LF characters (carriage return + line feed, or \r\n). These days, these programs work fine with either CR+LF or with LF alone. Meanwhile, Linux/Mac programs expect LF alone.

Use LF alone! There’s no reason to include the CR characters, even if you’re working on Windows.

One danger: new files created in programs like Notepad get CR+LF. Those files look like they have \r on every line when viewed in Linux/Mac programs or (in code) read into strings and split on \n.

That’s why, on Windows, it makes sense to ask git to change line endings from CR+LF to LF on files that it saves. core.autocrlf=input says, screw with the line endings only in one direction. Don’t add CR, but do take it away before other people see it.

Postscript

I love ternary booleans like this: true, false, input. Hilarious! This illustrates: don’t use booleans in your interfaces. Use enums instead. Names are useful. autocrlf=ScrewWithLineEndings|GoAway|HideMyCRs

Keynote: Collective Problem Solving in Music, Science, Art, and Software

(originally titled: “On the Origins of Opera and the Future of Programming”)

Blog write-up

Abstract

There’s a story to tell, about musicians, artists, philosophers, scientists, and then programmers.

There’s a truth inside it that leads to a new view of work, that sees beauty in the painful complexity that is software development.

Starting from The Journal of the History of Ideas, Jessica traces the concept of an “invisible college” through music and art and science to programming. Along the way she learns what teams are made of, where great developers come from, why changing software is harder for some of us than others, and why this is a fantastic time to be alive.

Past public deliveries

Layers in software: from data to value

Then

Back in the 2000s, we wrote applications in layers.

Presentation layer, client, data transfer, API, business logic, data access, database. We maintained strict separation bet ween these layers, even though every new feature changed all of them. Teams organized around these layers. Front end, back end, DBAs.

Each layer of software is a wide box, next to its team.
They stack on top of each other: frontend stuff, backend stuff, database, each with its team.
At the top are some customers. Value flows from them to the db and back, crossing all the layers.
Business value exists only by flowing through all the layers to the DB and back.

Layers crisscrossed the flow of data.

Responsibility for any one thing to work fell across many teams.

Interfaces between teams updated with every application change.

Development was slow and painful.

Now

In 2019, we write applications in layers.

A business unit is supported by a feature team. Feature teams are supported by platforms, tooling, UI components. All teams are supported by software as a service from outside the company.

Feature teams at the top of the software are multicolored, with multiple components in their software.
Under them are platform and component teams, each different.
Under them are nice square boxes of external services.
Business value flows through the top layer (feature teams), staying close to the business people.
Developer value flows between the feature teams, through the internal teams, to external services and back.
Business value is concentrated in the feature teams; developer value flows through support teams and external services.

Back in the day, front end, back end, operations, and DBAs separated because they needed different skills. Now we accept that a software team needs all the skills. We group by responsibility instead — responsibility for business value, not for activities.

Supporting teams provide pieces in which consistency is essential: UI components and some internal libraries.

Interfaces between teams change less frequently than the software changes.

Layers crisscross the flow of value.

DevEx

Feature teams need to do everything, from the old perspective. But that’s too hard for one team — so we make it easier.

This is where Developer Experience (DevEx) teams come in. (a.k.a. Developer Productivity, Platform and Tools, or inaccurately DevOps Teams.) These undergird the feature teams, making their work smoother. Self-service infrastructure, smooth setup of visibility and control for production software. Tools and expertise to help developers learn and do everything necessary to fulfill each team’s purpose.

Internal services are supported by external services. Managed services like Kubernetes, databases, queueing, observability, logging: we have outsourced the deep expertise of operating these components. Meanwhile, internal service teams like DevEx have enough understanding of the details, plus enough company-specific context, to mediate between what the outside world provides and what feature teams need.

This makes development smoother, and therefore faster and safer.

We once layered by serving data to software. Now we layer by serving value to people.