Reading: Why Information Grows

Yesterday in a zine, I read an in-process book review of The Rise and Fall of the Roman Empire. Reading that was a project for the review author; they wrote the review while still in the thralls of the book. That seems like the best time, not after I’ve closed the book. So quick! before I listen to the epilogue:

César Hidalgo is a statistical physicist. His book, Why Information Grows, is about biology and economics. It scales from the quantum to molecules to life to ecosystems to human systems. The common theme at every level is: information, embodied in matter. Processed by systems of increasing complexity to produce even more information. This is how our planet keeps being interesting, while the universe as a whole marches toward the boringness of maximum entropy.

Traditional thermodynamics says entropy always increases, OK, and then it uses this principle to predict the behavior of closed systems near equilibrium.

Which is no system ever.

There is only one closed system: the universe. And it is nowhere near equilibrium. When you get into a planet like Earth, which then develops feedback loops like self-replicating proteins, which develop into feedback loops like ecosystems, and eventually into human systems — we’re only moving farther from equilibrium. Energy is constantly injected into the system, and we use that to process and create higher and higher levels of information. More and more interesting stuff.

Hidalgo extends this into economics, and uses it to explain geographic income disparity.

The trick is: information by itself is useless. To use it, a system/entity/person needs enough knowledge to know what it means, and enough know-how to do something with it.

His key metaphor is that products are embodied information. To create any product takes knowledge and know-how. From a little, like garments, to a lot, like airplanes. That knowledge and know-how must exist inside a country for that country to make such products. And knowledge and know-how are way, way harder to move around than raw materials. They exist inside people, and networks of people, and networks of networks of people. So a person can make a rug, a firm can make a jersey, and many firms together can make a satellite.

Furthermore, Knowledge and know-how create knowledge and know-how. There’s a feedback loop: I know how some aspects of how to build software, so I build it, and I work with other people, and we learn from each other and the knowledge in our particular piece of the world increases. We also create new k&kh, both in ourselves, plus embodied in our software, which can give us and other people new tools to build new things. It spirals.

So software concentrates in Silicon Valley. Movies in Hollywood. Financial instruments in New York. Cars in Germany and Japan.

I’m also reading a book now, Cybernetic Revolutionaries, about a sophisticated cybernetic computer system built in Chile in 1971–72, a collaboration between many Chileans and a few key foreigners. Stafford Beer and some other British computer scientists worked with them on cybernetics, economic modelling, and programming. Guy Bonsiepe (German) worked with them on industrial design. The project and the whole country were greatly held back by US embargos which deprived them of parts and computers that Chile didn’t have the knowledge and know-how to build for themselves. The story illustrates the value of acquiring knowledge in a nation, and the limitations of not already having it.

Personally, this concept of knowledge and know-how being embodied in people — not in text! text only helps if you already have the knowledge to attach meaning to the text. And the way it transfers is by working together. By pair programming, not by documenting your code.

I can build a thing; we build the thing together; now you can build the thing (and better). We know this is how we learn best. Why do we pretend that text is sufficient?

There’s also the part where, he looks at cars or pillows as embodied information: products have the value that you can get benefit from the information without having to understand all of it. Software is very obviously that. The value in the software is what people can now do, that they couldn’t before without possessing a whole lot of knowledge and know-how that the creators of the software put into it. Note that here, software is a medium, and the value is in the business, in whatever users can do with it.

In the Three Body Problem series, one of the characters describes inhabited planets as “low-entropy systems,” and humans and aliens as “low-entropy beings.” We get there by using and creating information, by means of the knowledge and know-how embodied in the organization of our components — proteins, cells, neurons, employees, firms, nations. This book has me thinking hard about how knowledge is created and moved. Because it is not through books or the internet — printed language is the feeblest form of knowledge transfer. Working alongside each other is the richest. Can we do more of that?

Do Things Right with TypeScript

A collection of pointers for the present and future.

Print the whole error

`tsc –noErrorTruncation`

Why tsconfig.json is hard

tsconfig.json is important, because the TS compiler does way more, way more flexibly, than the Java compiler does. It’s both a transpiler and a typechecker.

What kind of JS do you want outputted? Choose your level of ECMAScript compatibility (as ancient as ES3 or as modern as ESnext) and also your module system compatibility (commonjs, amd, or several more).

What will be magically be available in your runtime? Bring in the type declarations for these things (such as the DOM) with the `”lib”: [“DOM”]` compiler option, or in `”types”: [“@types/node”]` (node module globals like `__filename`, or node built-ins like `fs`).

Also choose how stringent the typechecking is, with “strict” and its various suboptions.

Choose where your input files are, and where your output files go.

Choose what to output: only JS? sourcemaps? type declarations? type declaration maps?  … and for those maps, choose relative paths to the source.

The good news is: even if compilation has errors, tsc will output JS. So you can test even when tricky compile errors that you can’t figure out plague you.

Iterate through objects and arrays

There exists both `for (const a of array)` and a similar construct which shall not be named but contains the work `in` instead of `of`. Do not use that one.

To iterate through an array: for (const a of array) { … }

… ok I can’t stand it anymore I’m moving this post to Medium. Google clearly does not care about this platform, it may go away like Google Reader, and it is super painful to put code here.

Do Things Right with npm install

Lately I’ve been wrestling with npm. Here are some rules I’ve learned:

Use `npm ci` rather than `npm install`

`npm ci` will bring down exactly the dependencies specified in package-lock.json. `npm install` does more than that; it also tries to update some libraries to a more recent version. Sometimes it updates URLs or nonsense in package.json so that it my `git status` is dirty. Sometimes it does deduping. Sometimes it sticks the version you have lying around. I haven’t figured it out. It seems to be pretty dependent on the current circumstances on my file system.

Now I only use `npm install` if I specifically want to change the dependencies in my filesystem.

Use `npm install –save-exact`

Especially for snapshots. Semver does not work for snapshots or branches or anything but releases. And npm only works with semver. If you are not using a release; if you publish with build tags or branch tags or anything like that; do not give npm any sort of flexibility. It will not work. Specify a precise version or else it will give you nasty surprises, like deciding some alphabetically-later branch is better than the master-branch version you specified.

Use `npm view` to check the status of a library

This is quite useful. Try `npm view ` and it brings to your command line the info you can get from the npm website. You can ask it for specific fields. To get the latest version of chalk:

$ npm view chalk dist-tags.latest

If you want to do anything programmatic with this info, the “do things right” flag for `npm view` is `–json`.

Try `npm ls` but then dig around on the filesystem

Exploring the dependency tree, `npm ls` is really cool; it shows it to you. You can see where you’re getting a specific library with `npm ls ` except that it doesn’t always work. In the end, I dig around in my node_modules directory, using `find -name .` to look for the real thing.

Other times I use my little node-dependency dungeon explorer game to see what version of stuff is where. 

These are just a few of the nasty surprises I’ve found moving from Java to TypeScript, from maven dependencies to npm. Dependency management is an unsolved problem, and the people working on npm have made huge improvements in the last few years. I look forward to more.

the future of software: complexity

The other day in Iceland, a tiny conference on the Future of Software Development opened with Michael Feathers addressing a recurring theme: complexity. Software development is drowning in accidental complexity. How do we fight it? he asks. Can we embrace it? I ask.

Complexity: Fight it, or fight through it, or embrace it? Yes.

Here, find tidbits from the conference to advance each of these causes, along with photographs from a beautiful cemetery I walked through in Reykjavik.

Fight it

One way we resist complexity: keep parts small, by creating strong boundaries and good abstractions. Let each part change separately. The question is, what happens outside these boundaries?

Carefully bounded grave sites, each very different, each organized in its own way. You wish your software system was this pretty.

Feathers complained that developers spend about ten percent of their time coding, and the rest of it configuring and hooking together various tools. This makes sense to me; we’ve optimized programming languages and libraries so that coding takes less time, and we’ve built components and services so that we don’t have to code queuing or caching or networking or databases. Hooking these together is our job as software developers. Personally, I want to do more of that work in code instead of screens or configuration, which is part of my mission at Atomist. Josh Stella of Fugue also says we should be programming the cloud, not configuring it.

Paul Biggar at Dark has another way to attack complexity: wall it off. Cross the boundaries, so that developers don’t have to. Or as Jason Warner put it, “do a lot more below the waterline.” The Dark programming system integrates the runtime and the database and the infrastructure and the code, so that system developers can respond to what happens in the real world, and change the whole system at once. This opens backend development to a whole realm of people who don’t have time to learn the a dozen parts and their interconnections. The Future of Software Development is: more people will be doing it! People whose primary job is not coding.

In any industry, we can fight complexity through centralization. If everyone uses GitHub, then we don’t have to integrate with other source code management. Centralization is an optimization, and the tradeoffs are risk and stagnation. Barriers to entry are high, options are limited, and growth is in known dimensions (volume) not new ones (ideas).

Decentralization gives us choices, supports competing ideas, and prevents one company from have enough power to gain all the power. Blockchain epitomizes this. As Manuel Araoz put it: “Blockchain is adding intentional inefficiency to programming” in order to prevent centralization.

Centralization is also rear-facing: this thing we know how to do, let’s do it efficiently. Decentralization is forward-facing: what do we not yet know how to do, but could?

Building one thing very well, simply, is like building a stepping stone through the water we’re drowning in. But stones don’t flow. Exploration will always require living in complexity.

Fight through it

Given that complexity surrounds us, as it always will when we’re doing anything new, can we learn more ways to cope with it?

Plants and moss reach between grave sites. Complexity happens.

To swim forward in this complexity, we need our pieces to be discoverable, to be trouble-shoot-able, and to be experiment-with-able.

Keith Horwood from stdlib is working on the democratization of APIs, those pieces of the internet that make something happen in the real world. They’re making APIs easy to document and standardize. Stdlib aims to supplement, not replace developer workflows: the tools pile higher, and this is normal. Each tool represents a piece of detailed know-how that we can acquire without all the details.

Keith Horwood and Rishabh Singh both pointed out that humans/programmers go from seeing/reading, to speaking/writing, and then to executing/programming: we observe the world, we speak into the world, and then we change the world. (I would argue that we hop quickly to the last step, both as babies and as developers.) To learn how to change a complex system is to change it, see what happens, change it again.

We use type systems and property tests to reason about what can’t happen. Example tests and monitoring reassure us what happens in anticipated circumstances. We get to deduce what really does happen from observability and logs.

Embrace it

If we accept that we are part of this complex system that includes our code, perhaps we can find buoyancy: we can sail instead of drown.

Chaos is not unordered. It took us a long time to understand its order.

Complexity is not anarchy; when it gels, a complex system is a higher form of order. It is an order that operates not in linear deductions, but in circles and spirals. These circles operate both within the system and between the system and its environment.

Feathers and I both spoke about the symbiosis of a development team with its code and its tools. I call it a symmathesy. We learn from our code, from the clues it leaves us in data and logs; and our code learns from us, as we change it. Both these forms of communication happen only through other software: observability tools to see what is happening in the software, and delivery tools to change what will happen. Once we view the system at this level, we can think about growing our whole team: people, running software, tools for visibility and control.

Rishabh Singh, Miltos Allamanis, and Eran Yahav showed machine-learning backed tooling that makes programs that offer useful suggestions to humans who are busy instructing the computer. The spiral goes higher.

Kent Beck said that nothing has higher leverage than making a programming system while using those same tools to build a system. His talk suggested that we: (1) make very small changes in our local system; (2) let those changes propagate outwards gradually; and (3) reflect on what happened, together. We learn from the system we are changing, and from each other.

McLuhan’s Law: We shape our tools, and then our tools shape us.

Our tools don’t shape our behavior violently and inflexibly, the way rules and punishment do. They shape us by changing the probability of each behavior. They change what is easy. This is part of my mission at Atomist: enable more customization of our own programming system, and more communication between our tools and the people in the symmathesy.

a cute little idiomatic grave site

As developers, we are uniquely able to shape our own world and therefore ourselves by changing our tools. Meanwhile, other people are gaining some of this leverage, too.

I believe there will be a day when no professional says “I can’t code” — only “coding is not my specialty.” Everyone will write small programs, what Ben Scofield calls idiomatic software, what I call personal automation. These programs will remain entwined with their author/users; we won’t pretend that the source code has value outside of this human context. (Nathan Herald had a good story about this, about a team that tried to keep using a tool after the not-professional-developer who wrote it left.)

This is a problem in development teams, when turnover is high. “Everyone who touches it is reflected in the code.” (who said that? Rajeev maybe?) I don’t have a solution for this.

a smooth path through the cemetery

The path forward includes more collaboration between humans and computers, and between computers and each other, guided by humans. It includes building solid steps on familiar ground, swimming lessons for exploration, and teamwork in the whole sociotechnical system so that we can catch the winds of complexity and make them serve us.

Get happy developers to do what you want

(originally published in the SD Times)

How do we influence the operation of a complex dynamical system like a team? How do we get a development team to function closer to our preferences, and still let people operate as self-actualized humans?

In Dynamics in Action, Alicia Juarrero describes complex systems such as people in terms of the constraints they operate within. Constraints make things interesting, and constraints allow for emergence.

Constraints here aren’t hard rules. They aren’t restraints, holding people still. Rather, constraints are changes in the probability of particular actions.
They’re like … an incentive landscape.

For instance: if you force people to change their password frequently, they’ll use a terrible password. If coffee is free and juice costs money, people will drink more coffee. If the stairs are pretty and easy to access, people will take them more often than the elevator.

These are all ambient (context-free) constraints. They don’t restrain the team from taking the elevator, but they do change the probability of whether each person, on each occasion, will do so.

restrain: one allowable path. constrain: one prettiest path

As a software organization — maybe as an architect, or technical lead, or SRE, or infrastructure team — how do you get developers to code in a certain way? to be consistent with the rest of the organization, to follow your 
preferred practices? Getting developers to do anything is supposed to be like herding cats.

Change the cost, change the behavior.

Development is a creative undertaking. Restraints don’t sit well. But if you make something easier for me, that’s a constraint I can live with. It will change my probabilities.

I do this for myself: a script to make submitting a single change as a PR into one step, and now I less frequently lump that function rename in with other unrelated changes.

My team expects me to look at every change coming in to the repositories we work on together, in many time zones. That seemed onerous, until I made my “git fetch and move my default branch pointer” script also open a GitHub compare window. Now this feels normal, and I wouldn’t work without this routine.

When we work in computers, a new automation changes our constraints, and this changes the probabilities of our various behaviors.

I can affect other people in my organization: I want people to improve the descriptions in our GraphQL schema, so I made an automation that deploys the schema when they push a button in Slack. I made the button appear whenever someone changes the schema in GitHub. They use this all the time.

When my coworker wants the dependencies alphabetized: sorry, I don’t care enough. He can add an autofix — a TypeScript function in our software delivery machine — to do that.

I can change the incentive landscape by making the way I want people to make software, the easy way to make software. Sometimes by giving them a button, other times having a robot come behind and clean up every push with an autofix. The autofix has the added benefit of keeping the change I want in the system, which is often harder than making the change.

Contrast this with code metrics as an incentive: metrics distract from our collective goal of making this particular software great. They draw our attention to numbers, instead of customers. Metrics are great for information, but as soon as they’re used to alter the incentive landscape, they distort it in unintended ways.

Whether for yourself or the other developers in your organization, consider automation a tool to constrain (influence) the way developers work, without curtailing freedom. Increase developer happiness and consistency at the same time!