In business, we want to focus on our core domain, and let everything else be “good enough.” We need accounting, payroll, travel. But we don’t need those to be special if our core business is software for hospitals.
As developers, we want to focus on changing our software, because that is our core work. We want other stuff, such as video conferencing, email, and blog platforms to be “good enough.” It should just work, and get out of our way.
The thing is: “good enough” doesn’t stay good enough. Who wants to use Concur for booking travel? No one. It’s incredibly painful and way behind modern web applications that we use for personal travel. Forcing them into an outdated travel booking system holds your people back and makes recruiting a little harder.
When we rent software as a service, then it can keep improving. I shuddered the last time I got invited to a WebEx, but it’s better than it used to be. WebEx is not as slick as Zoom, but it was fine.
There is a lot of value in continuing with the same product that your other systems and people integrate with, and having it improve underneath you. Switching is expensive, especially in the focus it takes. But it beats keeping the anachronism.
DevOps says, “If it hurts, do it more.” This drives you to improve processes that are no longer good enough. Now and then you can turn a drag into a competitive advantage. Now and then, like with deployment, you find out that what you thought was your core business (writing code) is not core after all. (Operating useful software is.)
Limiting what you focus on is important. Let everything else be “good enough,” but check it every once in a while to make sure it still is. Ask the new employee, “What around here seems out of date compared to other places you’ve worked?” Or try a full week of mob programming, and notice when it gets embarrassing to have six people in the same drudgery.
Here’s a lovely graphic from the folks at Intercom. It describes the difference between what companies sell and what people buy.
We don’t want the tool, we want what we can do with the tool. Take it further – maybe what that skateboarder really wants is: the high fives at the end.
Our accomplishments feel real when people around us appreciate them. If the skateboarder’s peers react with “Why would you do that?? You could die! You look like a dumbass,” titanium hardware doesn’t shine so bright.
It reminds me of the DevOps community. Back when coworkers said, “Why are you writing scripts for that? Do you want to put us out of a job?” automation wasn’t so cool. Now there’s a group of peers, at least online, ready to give you high fives for that. The people you meet at conferences, or get familiar with on podcasts, or collaborate on open source tools with — these take automation from laziness into accomplishment.
Giving developers polyurethane wheels and hollow trucks won’t let them do tricks. Move to a culture of “our job is to automate ourselves out of better and better jobs.” Give us the high fives (and let us pick our tools), and developers will invent new tricks.
Symmathecist: (sim-MATH-uh-sist) an active participant in a symmathesy.
A symmathesy (sim-MATH-uh-see, coined by Nora Bateson) is a learning system made of learning parts. Software teams are each a symmathesy, composed of the people on the team, the running software, and all their tools.
The people on the team learn from each other and from the running software (exceptions it throws, data it saves). The software learns from us, because we change it. Our tools learn from us as we implement them or build in them (queries, dashboards, scripts, automations).
This flow of mutual learning means the system is never the same. It is always changing, and its participants are always changing.
An aggregate is the sum of its parts. A system is also a product of its relationships. A symmathesy is also powered by every past interaction.
I aim to be conscious of these interactions. I work to maximize the flow of learning within the system, and between the system and its environment (the rest of the organization, and the people or systems who benefit from our software). Software is not the point: it is a means, a material that I manipulate for the betterment of the world and for the future of my team.
I am a symmathecist, in the medium of software.
my “Origins of Opera” keynote (video or summary) introduces this term
Software Development has moved forward a lot recently. Both on the code side and the runtime side, we’ve had huge advances.
Then on the runtime side, we don’t have to deploy to hardware anymore. Virtual machines were a step, and then the cloud, and now Kubernetes is a big deal. Kubernetes is a big deal because it’s higher level of abstraction, where developers can work in the language of the domain of software. But that’s not all: Kubernetes also offers an API, which means we can work with it using code.
We can do more with code now on both sides, thanks to expressive frameworks and to an API for hardware. But there’s something in the middle lagging behind.
The bridge from source code to running software is Delivery. Delivery has made advances: we went from Bash and make to pipelines. But pipelines have been around for a decade, and since then we’ve got … morepipelines. It’s time for the next revolutionary step.
This is Atomist’s mission: to change the way we deliver software. Not to make it incrementally better, but to rethink delivery the way we’ve rethought application architecture and runtime infrastructure.
The way forward is not more pipelines. Nor is it event more bash scripts or configuration that wishes it were code. The way forward is not updated separately for each delivered application.
The way forward is to do more with code. In a real programming language. The way forward responds to events in the domain of software delivery: to code pushes, and also tickets, pull requests, deploys, builds, and messages in chat. It responds in context: when our delivery machine decides (in code!) what to do with a particular change, it does so with awareness of what the code says, of who changed it, in response to what ticket. It responds in communication: when in doubt, contact the responsible human and ask them whether they’d like to restart the build, submit the pull request, or deploy to production.
As we succeed, our systems increase in complexity. The systems to control them need to be at least as smart. We need more power in our delivery than any GUI screen or YAML file can give us. We have that power, when we craft our delivery in code on a strong framework with a domain-specific API.
Every company is in the software delivery business now. Let’s take it seriously, the same way we do our code and our production runtime.
Yesterday I was ready to get some changes into master, so I merged in the latest and opened a PR. But NO, the build on my pull request broke. The error was:
ERROR: (jsdoc-format) /long/path/to/file.ts[52, 1]: asterisks in jsdoc must be aligned ERROR: (jsdoc-format) /long/path/to/file.ts[53, 1]: asterisks in jsdoc must be aligned ERROR: (jsdoc-format) /long/path/to/file.ts[54, 1]: asterisks in jsdoc must be aligned ERROR: (jsdoc-format) /long/path/to/file.ts[55, 1]: asterisks in jsdoc must be aligned … fifty more like that …
gah! someone added more tslint rules, and this one doesn’t have an automatic fix. Maybe someone upgraded tslint, its “recommended” ruleset changed, and bam, my build is broken.
For measley formatting trivia.
/** * oh no! This JSDoc comment is not aligned perfectly! * The stars are supposed to have one more space before them * so they all line up under the first top one * * The world will end! Break the build! * * @param likeItMatters a computer could fix this grrr * @deprecated */ whatever(likeItMatters: Things): Stuff;
Look, I’m all for consistent code formatting. But I refuse to spend my time adding a space before fifty different asterisks. Yes I know I can do this with fewer keystrokes using editor acrobatics. I refuse to even open the file.
You know who’s good at consistency? Computers, that’s who. You want consistent formatting? Make a robot do it.
So I made a robot do it. In our Software Delivery Machine, we have various autofixes: transformations that run on code after every commit. If an autofix makes a change, the robot makes a commit like a polite person would. No build errors, just fix it thanks.
I wrote this function, which does a transformation on the code. The framework takes care of triggering, cloning the repository, and committing the result.
You can do this too. You can do it on your local machine with the fully open-source Local Software Delivery Machine. Fix your code on demand, or in response to each commit. Write your own functions to keep your code looking the way you like. Never be embarrassed by the same mistake again!
To help you try it out, I added my autofix to an otherwise-empty Software Delivery Machine.
Go to that location: cd $ATOMIST_ROOT/atomist-blogs/align-stars-sdm
Now for the slow part: npm install
Start up your SDM: atomist start --local The SDM is a process that hangs out on your computer waiting to help. It swings into action when triggered by the atomist command line, or by commits inside $ATOMIST_ROOT.
Optional: in another terminal, run atomist feed. This will give you a running summary of what your SDM is up to.
Now screw up some formatting. I’ve left some nice jsdoc comments in lib/autofix/alignStars.ts; move those stars over a little.
Save and make a commit: git commit -am “Oh no, misalignment”
Check the output in your atomist feed, and you should see that an Autofix has run. (You can also type atomist align the stars to do this specific transform, in a repository not wired up to trigger you SDM.)
Check your commit history: git log. Did Atomist make a commit? Check its contents: git show and you should see the stars moved into alignment.
OK that was a lot just to format some comments. The important part is we can write functions to realize policy. If a person complained about this formatting I’d tell them to f*** off or fix it themselves — in fact I did curse extensively at tslint when it gave me these errors. Tslint didn’t care. Programs aren’t offended.
If I want my teammates’ code to conforms to my standards, it’s rude to ask them to be vigilant about details. Aligning asterisks — I mean, some people like it, and that’s fine, I kinda enjoy folding laundry — but as a developer that’s a crummy use of my attention. Computers, though! Computers are great at being vigilant. They love it. My Atomist SDM sits there eagerly awaiting commits just to dig around in those asterisks in the hope of fixing some.
Please make my star-alignment into a code transform that’s useful to you. I went with plain string parsing here, in a very functional style for my own entertainment. We also have clever libraries for working with the compiled AST and more (watch this space).
There’s more: an SDM running in the cloud listens to GitHub (GitLab, BitBucket, GHE) and applies autofixes to everyone’s commits. And code reviews. And runs or triggers build (but only when it’s worthwhile; it looks at what’s changed). And initiates deploys (except it asks us first in Slack). There’s no setting up a pipeline for a new repository or branch; our SDM knows what to do with a code push based on the code in it.
There’s more: an SDM in the cloud also listens to issues, pull requests, builds, deploys, and other events in the software domain. It can react to all of them by talking to people in Slack, or running any other program. Whatever we do that is boring, we can automate.
This is our power as software developers. We don’t need someone to write a GUI we can click in. We don’t need to configure in YAML. We can specify what needs to happen declaratively in code.
All we needed was a framework to do the common glue-y bits for us, like triggering on events, cloning repositories, passing our functions the data they need and carrying out our actions.
The autofix in this example triggers on commit, clones the repository, passes a Project object in for a code transform function to act on, commits those changes and pushes them back to where I’m working. The framework of the SDM lets me define my own policies and implement them in code. Then the machinery of the SDM keeps them running, locally (open source) or team- or organization-wide (using Atomist’s API).
Building a new automation to give power to my team
At Atomist, we supply a GraphQL endpoint to reveal links between commits, builds, deploys, and people. Internally, we also use GraphQL to access data in Neo4J. To do that, we need to deploy a schema definition (IDL) to the Neo4j GraphQL extension.
Software evolves from simple to rigorous. In the experimental stage (“do we want to use this?”) I constructed a schema.idl and pushed it to Neo4J by hand. One person doing all the schema is okay for a little while, but not for production. I want other people to improve the schema, too! So I automated its path from a GitHub commit to the database. Now any member of our team can deploy a new schema, and they’re even prompted to do it in Slack after they push a change to schema.idl.
This automation lives in an Atomist automation client, which turns a program I could run locally into one my whole team can run. I pushed it up to the cloud, and now the whole team has my power to deploy the schema.
First, I made a command that retrieves the file from GitHub and posts it to Neo4J. The passwords and URLs are all in one place (my automation deployment), so people don’t need their own database login to upgrade the schema. The command can be triggered in Slack, or from a REST API (web form coming soon). Now everyone can deploy the schema.
This replaces two account creations, three logins, two curl commands, and some cryptic JSON interpretation with one “@atomist deploy graphql schema” message in Slack. Then I made it even better.
Next, I made an automation that recognizes when the schema.idl file changes, since that one is significant. It reports that significance back to Atomist, and this shows up in Slack as part of the standard Atomist report about commits pushed to GitHub. This is a way of noticing which commits affect different parts of the system.
Finally, another automation responds to that change in schema.idl and pops a Slack message into the channel with a button! Push the button, and that “deploy graphql schema” command updates the schema in Neo4J, either the staging database (if you pushed the change to a branch) or production (if it’s mainline). Super convenient!
This small piece of infrastructure matures our internal GraphQL implementation; it’s trackable, repeatable, safe, and quick. It’s easier for me, and a new power for the rest of the team — they didn’t even bother touching it before, and now they can. I love turning complicated READMEs into happy, timely buttons in Slack!
You can make a big piece of software more mature with a small automation.
Find the code for my automation here. If you’d like to hear more details about how to implement this example, so that you can do this kind of automation on your team, clap for this post or ping me in the Atomist community Slack. (Or get started right away with a Quick Start.) I’d love to hear from you.
Tradeoffs are a fact of life. Choose between safety and speed, stability and flexibility. More of one sacrifices the other.
I remember a time when this described our decisions about software deployments. Deployments caused problems, so we did fewer of them.
Then came Continuous Integration and Continuous Deployment. With a lot of work, we get automated tests and monitoring, automated deployment and rollback. Then we learned that small deployments are safer — so much safer that many fast deployments added up to less danger than one bulk deployment.
Fast and Safe
It’s like, this automation plus culture change has folded space — these arrows now point in the same direction.
Bending tradeoff space like this wasn’t free.
Tests double our application code, and infrastructure adds even more code. Harder: we had to change the culture. Think in smallest shippable units. Alleviate the next bottleneck after deploys aren’t it, and the next.
Without weeks of manual testing, we had to create new ways to answer “Does it work?” Without ops people standing by, we needed new ways to answer “is it up?” When each person makes deployment decisions individually, they need tools to support the decision with visibility, and tools to implement the decision with fast rollback in case of surprise.
This automation is expensive, but we wouldn’t go back to the old days of dangerous, slow deployments.
Flexible v Consistent
I want to talk about another tradeoff space, this one in “Do we create custom internal software?” As a business, I want internal software to increase the consistency, legibility, and speed of my employees’ work. We think of the cost of software in terms of money and time, work to create it and to keep it happy. There’s a deeper cost to the organization, though: ossification.
The consistency enforced by software is great for managers and administrators; they get legibility, meaning they can track work at a high level. These are the people making decisions about software requirements.
I once worked on workflow tracking software for a research organization. The heads of the development effort bragged that they’d encoded the knowledge of the scientists, so they could leave and we’d still have it. The research could proceed… in exactly the same way. Any substantial change took (this is extreme, but it was ten years ago) eighteen months to roll out. So much for scientific innovation! oh but it was great for the higher-ups tracking progress of each experiment as it moved through this regimented system.
This consistency is not optimal for learning, not for daily patient care or customer service. People are adaptable, but software inhibits this adaptation. Ask any nurse, social worker, or hotel receptionist how they feel about the software they have to use. (Ask them when they are off duty.) They’ll tell you about frustrations, time wasted, obstacles to doing great work.
Compared to a paper form, an electronic form is constraining. Compared to talking to each other, coordinating via task trackers is limiting. Software ossifies processes. It impedes learning and improvisation.
Flexible and Consistent
How could we fold this tradeoff space?
This tradeoff exists when the people using the software can’t change the software. Maybe it can help to look at one place where this tradeoff does not exist, where it is naturally folded.
When the users are also the system movers, calcification doesn’t happen.
Software teams developing for themselves gain consistency, legibility, speed and more — while adding flexibility. When deployment is automated in one place, we can improve the process without the work of disseminating information.