Don’t just keep trying; report your limits

The other day we had problems with a service dying. It ran out of memory, crashing and failing to respond to all open requests. That service was running analyses of repositories, digging through their files to report on the condition of the code.

It ran out of memory trying to analyze a particular large repository with hundreds of projects within it. This is a monorepo, and Atomist is built to help with microservices — code scattered across many repositories.

This particular day, Rod Johnson and I paired on this problem, and we found a solution that neither of us would have found alone. His instinct was to work on the program, tweaking the way it caches data, until it could handle this particular repository. My reaction was to widen the problem: we’re never going to handle every repository, so how do we fail more gracefully?

The infrastructure of the software delivery machine (the program that runs the analyses) can limit the number of concurrent analyses, but it can’t know how big a particular one will be.

However, the particular analysis can get an idea of how big it was going to be. In this case, one Aspect finds interior projects within the repository under scrutiny. My idea was: make that one special, run it first, and if there are too many projects, decline to do the analysis.

Rod, as a master of abstraction, saw a cleaner way to do it. He added a veto functionality, so that any Aspect can declare itself smart enough to know whether analysis should continue. We could add one that looks at the total number of files, or the size of the files.

We added a step to the analysis that runs these Vetoing Aspects first. We made them return not only “please stop,” but a reason for that stop. Then we put that into the returned analysis.

The result is: for too-large repositories, we can give back a shorter analysis that communicates: “There are too many projects inside this repository, and here is the list of them.” That’s the only information you get, but at least you know why that’s all you got.

And nothing else dies. The service doesn’t crash.

When a program identifies a case it can’t handle and stops, then it doesn’t take out a bunch of innocent-bystander requests. It gives a useful message to humans, who can then make the work easier, or optimize the program until it can handle this case, or add a way to override the precaution. This is a collaborative automation.

When you can’t solve a problem completely, step back and ask instead: can I know when to stop? “FYI, I can’t do this because…” is more useful than OutOfMemoryError.

Design by implementation

We can develop things faster than we can define them.

@jryanday

I’ve remarked before that @cdupuis can implement new features way faster than I can document them. This is not ridiculous; creating something is often faster than connecting it to existing people who can use it.

Ryan took this farther: we can’t even define what we want faster than we can implement it, sometimes. This is not ridiculous; design informs implementation, and implementation informs design.

Implementation informs design, and the process of implementation builds a mental model in the developer’s head. Making that model concrete and communicating it to someone else is hard, sometimes harder than getting the code working. Communicating that model in a way that oodles of people can use it is harder still, sometimes an order of magnitude harder. It includes documentation, yes, and marketing and customer service and advocacy. If you’re really serious, writing a specification. These all inform the next iterations of design and of implementation.

Developing things is only a beginning.

Align the stars (programmatically)

Yesterday I was ready to get some changes into master, so I merged in the latest and opened a PR. But NO, the build on my pull request broke.
The error was:

ERROR: (jsdoc-format) /long/path/to/file.ts[52, 1]: asterisks in jsdoc must be aligned
ERROR: (jsdoc-format) /long/path/to/file.ts[53, 1]: asterisks in jsdoc must be aligned
ERROR: (jsdoc-format) /long/path/to/file.ts[54, 1]: asterisks in jsdoc must be aligned
ERROR: (jsdoc-format) /long/path/to/file.ts[55, 1]: asterisks in jsdoc must be aligned
… fifty more like that …

gah! someone added more tslint rules, and this one doesn’t have an automatic fix. Maybe someone upgraded tslint, its “recommended” ruleset changed, and bam, my build is broken.

For measley formatting trivia.

 /**
* oh no! This JSDoc comment is not aligned perfectly!
* The stars are supposed to have one more space before them
* so they all line up under the first top one
*
* The world will end! Break the build!
*
* @param likeItMatters a computer could fix this grrr
* @deprecated
*/
whatever(likeItMatters: Things): Stuff;

Look, I’m all for consistent code formatting. But I refuse to spend my time adding a space before fifty different asterisks. Yes I know I can do this with fewer keystrokes using editor acrobatics. I refuse to even open the file.

You know who’s good at consistency? Computers, that’s who. You want consistent formatting? Make a robot do it.

tslint robot: “Your Code Is Not Acceptable!” / me: “I have better things to do :-(” / Atomist: “I will save you!”

So I made a robot do it. In our Software Delivery Machine, we have various autofixes: transformations that run on code after every commit. If an autofix makes a change, the robot makes a commit like a polite person would. No build errors, just fix it thanks.

I wrote this function, which does a transformation on the code. The framework takes care of triggering, cloning the repository, and committing the result.

const alignAsterisksInProject: CodeTransform = (project: Project) =>
doWithFiles(project, "**/*.ts", async f => {
const content = await f.getContent();
if (hasUnalignedAsterisks(content)) {
await f.setContent(alignStars(content));
}
});

You can do this too. You can do it on your local machine with the fully open-source Local Software Delivery Machine. Fix your code on demand, or in response to each commit. Write your own functions to keep your code looking the way you like. Never be embarrassed by the same mistake again!

To help you try it out, I added my autofix to an otherwise-empty Software Delivery Machine.

Align your own stars

Try it yourself (on Mac or Linux, currently):

  • Install the atomist command line: npm install -g @atomist/cli@next
  • By default, atomist works on projects in ~/atomist/projects. If you prefer a different directory, create one and set ATOMIST_ROOT=/path/to/that/directory
  • Bring down the code for the Software Delivery Machine (SDM): atomist clone https://github.com/atomist-blogs/align-stars-sdm 
    (This does git clone, plus sets up git hooks for magical autofixes.)
  • Go to that location: cd $ATOMIST_ROOT/atomist-blogs/align-stars-sdm
  • Now for the slow part: npm install
  • Start up your SDM: atomist start --local
    The SDM is a process that hangs out on your computer waiting to help. It swings into action when triggered by the atomist command line, or by commits inside $ATOMIST_ROOT.
  • Optional: in another terminal, run atomist feed. This will give you a running summary of what your SDM is up to.
  • Now screw up some formatting. I’ve left some nice jsdoc comments in lib/autofix/alignStars.ts; move those stars over a little.
  • Save and make a commit: git commit -am “Oh no, misalignment”
  • Check the output in your atomist feed, and you should see that an Autofix has run. (You can also type atomist align the stars to do this specific transform, in a repository not wired up to trigger you SDM.)
  • Check your commit history: git log. Did Atomist make a commit? Check its contents: git show and you should see the stars moved into alignment.
  • (if not, please leave me an issue or ping me (jessitron) in community slack)

But wait! That’s not all.

OK that was a lot just to format some comments. The important part is we can write functions to realize policy. If a person complained about this formatting I’d tell them to f*** off or fix it themselves — in fact I did curse extensively at tslint when it gave me these errors. Tslint didn’t care. Programs aren’t offended.

If I want my teammates’ code to conforms to my standards, it’s rude to ask them to be vigilant about details. Aligning asterisks — I mean, some people like it, and that’s fine, I kinda enjoy folding laundry — but as a developer that’s a crummy use of my attention. Computers, though! Computers are great at being vigilant. They love it. My Atomist SDM sits there eagerly awaiting commits just to dig around in those asterisks in the hope of fixing some.

Please make my star-alignment into a code transform that’s useful to you. I went with plain string parsing here, in a very functional style for my own entertainment. We also have clever libraries for working with the compiled AST and more (watch this space).

There’s more: an SDM running in the cloud listens to GitHub (GitLab, BitBucket, GHE) and applies autofixes to everyone’s commits. And code reviews. And runs or triggers build (but only when it’s worthwhile; it looks at what’s changed). And initiates deploys (except it asks us first in Slack). There’s no setting up a pipeline for a new repository or branch; our SDM knows what to do with a code push based on the code in it.

There’s more: an SDM in the cloud also listens to issues, pull requests, builds, deploys, and other events in the software domain. It can react to all of them by talking to people in Slack, or running any other program. Whatever we do that is boring, we can automate.

This is our power as software developers. We don’t need someone to write a GUI we can click in. We don’t need to configure in YAML. We can specify what needs to happen declaratively in code.

All we needed was a framework to do the common glue-y bits for us, like triggering on events, cloning repositories, passing our functions the data they need and carrying out our actions.

The autofix in this example triggers on commit, clones the repository, passes a Project object in for a code transform function to act on, commits those changes and pushes them back to where I’m working. The framework of the SDM lets me define my own policies and implement them in code. Then the machinery of the SDM keeps them running, locally (open source) or team- or organization-wide (using Atomist’s API).

Aligning the stars is only the beginning.

Video: Automating at a Higher Level with Atomist

Philly ETE has published my video on InfoQ!

 This is about development automation: why it’s a great idea, why we should do more of it, and how Atomist is enabling it. There is live coding, there is me getting excited, and there is an awesome sparkly dinosaur shirt. I love this shirt. I also love automation, can you tell?

Video:
https://www.infoq.com/presentations/atomist

The talk is from April 2017. Abstract:

As developers, we automate. We automate other work, and sometimes we automate our own. We save typing with templates or IDEs. We save searching out information with Slack integrations. Many companies have custom internal bots to tie together chat, version control, and build servers.

Most companies can’t and shouldn’t dedicate whole teams to streamlining development. And they shouldn’t have to. Atomist is building programmable developer automation, from typing code to tracking status. Keep your code up-to-date with automated pull requests and generators. Keep yourselves up-to-date with mini-dashboards in chat that correlate commits with builds, complete with action buttons to cut down your context switches.

This session will demonstrate the standard Atomist coordination and automation tools, plus how to program instant automation for your code and team.

a Rug story: adding test cases

These days I work on Rug, Atomist’s library for coding code modifications.

Adding a feature, I start by creating a test. While it’s tempting to create a narrow test around the piece of code I want to change, it’s better to create an API-level test. Testing at the outside has a few benefits: it tells the story of why this feature is needed; it drives pleasing API design; and it places minimum constraints on the implementation. The cost is, it’s more work.

The API level of Rug is in TypeScript, where people write programs to modify other programs. The test compiles the TypeScript to JavaScript, and rug executes that inside the JVM, where our Scala code does the tricky work of implementing the Rug programming model — navigating the project, parsing code, and making atomic modifications (it all works or none of it is saved). This means that my API-level tests include a TypeScript file and a Scala file, plus a bunch of wiring to hook them together. I get tired of remembering how to do this. Plus, we’re constantly improving the programming model in TypeScript, so “the right way” is a shifting target.

Last year, I would have copied an existing test (which one is up to date? I don’t know! guess and hope it works), modified parts of it for my needs (and forgotten some), embedded the TypeScript code as a string in Scala (seems easier than making a .ts file), and tried to abstract away some of the repetitive bits that are shared between tests (even though that obscures the storyline of the test).

This year, I have a new tool. About the third time I needed a new test, I wrote a program to create it for me. I wrote a Rug! My AddTypeScriptTest Rug editor creates a new TypeScript file in test/resources, and a new Scala file in test/scala. It bases these off of sample files that exemplify the current standard in Rugs and their tests, performing all the modifications that I mess up in the copy-paste-modify strategy.

me:

rug edit -l AddTypeScriptTest class_under_test=com.atomist.rug.NewFeature

my Rug program:

  • copies SampleTypeScriptTest.scala to a new location. Changes the package name, the class name, and the location of the TypeScript file it will load.
  • copies SampleTypeScriptTest.ts to a new location. Changes the name of the class and the exported instance.

SampleTypeScriptTest.scala and SampleTypeScriptTest.ts form a real test in rug’s test suite, so I know that my baseline continues to work. When I update the style of them (as I did today), I can run the sample test to be sure it works (caught two errors today). I maximize their design to best tell the story of how rug goes from a TypeScript file to a Rug archive to running that program on a separate project and seeing the results. This helps people spinning up on Rug understand it. Repetition (of the Scala package name and the path to the test program, for instance) doesn’t hurt because a program is modifying them consistently (bonus: IntelliJ will ctrl-click into the referenced file on the classpath. It didn’t when that repetition was abstracted). If I want to change the way all these tests work, I can do that with a Rug editor too, since they’re consistent. Ahhhh the consistency: when a test breaks, and it looks exactly like the other tests except for meaningful differences, debugging is easier.

I created this Rug editor inside the rug project itself, since it’s only relevant to this particular project. Then I run the rug CLI in local mode, on the local project, and poof. I’ve used rug to modify rug using a Rug inside rug. Super meta! (It doesn’t have to be so incestuous. Other days, I use rug to modify any project using a Rug in any Rug archive.)

If you want to create a Rug to automate your own frequent tasks, install the Rug CLI and, from your project root, use this Rug: rug edit atomist-rugs:rug-editors:AddLocalEditor editorName=WhatDoYouWantToCallIt . Find your starting point in .atomist/editors/WhatDoYouWantToCallIt.ts

Pop into Atomist community slack with questions and we will be soooo happy to help you.

Today’s Rug: maven executable jar

Today’s Rug: maven executable jar

this is supposed to be a rug with a jar on it. An active jar.

I like being a polyglot developer, even though it’s painful sometimes. I use lots of languages, and in every one I have to look stuff up. That costs me time and concentration.

Yesterday I wanted to promote my locally-useful project from “I can run it in the IDE” to “I can run it at the command line.” It’s a Scala project built in maven, so I need an executable jar. I’ve looked this up and figured this out at least twice before. There’s a maven plugin you have to add, and then I have to remember how to run an executable jar, and put that in a script. All this feels like busywork.

What’s more satisfying than cut-and-pasting into my pom.xml and writing another script? Automating these! So I wrote a Rug editor. Rug editors are code that changes code. There’s a Pom type in Rug already, with a method for adding a build plugin, so I cut and paste the example from the internet into my Rug. Then I fill in the main class; that’s the only thing that changes from project to project so it’s a parameter to my editor. Then I make a script that calls the jar. (The script isn’t executable. I submitted an issue in Rug to add that function.) The editor prints out little instructions for me, too.

$ rug edit -lC ~/code/scala/org-dep-graph MakeExecutableJar main_class=com.jessitron.jessMakesAPicture.MakeAPicture
...
run `mvn package` to create an executable jar
Find a run script in your project’s bin directory. You’ll have to make it executable yourself, sorry.
...

→ Changes
├── pom.xml updated 2 kb
├── pom.xml updated 2 kb
├── bin/run created 570 bytes
└── .atomist.yml created 702 bytes

Successfully edited project org-dep-graph

It took a few iterations to get it working, probably half an hour more than doing the task manually. It feels better to do something permanently than to do it again. Encoded in this editor is knowledge: * what is that maven plugin that makes an executable jar? [1] * how do I add it to the pom? [2] * what’s the maven command to build it? [3] * how do I get it to name the jar something consistent? [4] * how do I run an executable jar? [5] * how do I find the jar in a relative directory from the script? [6] * how do I get that right even when I call the script from a symlink? [7]

It’s like saving my work, except it’s saving the work instead of the results of the work. This is going to make my brain scale to more languages and build tools.

— — — — — — — — — — — — — — — — — — — — — —

below the fold: the annotated editor. source here, instructions here in case you want to use it -> or better, change it -> or even better, make your own.

@description “teaches a maven project how to make an executablejar” @tag “maven” editor MakeExecutableJar
@displayName “Main Class”
@description “Fully qualified Java classname”
@minLength 1 @maxLength 100
param main_class: @java_package
let pluginContents = “””<plugin><groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<outputFile>target/executable.jar</outputFile>[4]
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals><goal>shade</goal></goals>
<configuration>
<transformers>
<transformer implementation=”org.apache.maven.plugins.shade.resource.ManifestResourceTransformer”>
<mainClass>__I_AM_THE_MAIN__</mainClass>
</transformer> </transformers> </configuration> </execution> </executions>
</plugin>“”” [2]
let runScript = “””#!/bin/bash
# http://stackoverflow.com/questions/59895/getting-the-source-directory-of-a-bash-script-from-within
SOURCE=”${BASH_SOURCE[0]}”
while [ -h “$SOURCE” ]; do
DIR=”$( cd -P “$( dirname “$SOURCE” )” && pwd )”
SOURCE=”$(readlink “$SOURCE”)”
[[ $SOURCE != /* ]] && SOURCE=”$DIR/$SOURCE”
done [7]
DIR=”$( cd -P “$( dirname “$SOURCE” )” && pwd )” [6]
java -jar $DIR/../target/executable.jar “$@” [5]
“””
with Pom p
do addOrReplaceBuildPlugin “org.apache.maven.plugins” “maven-shade-plugin” pluginContents [1]
with File f when path = “pom.xml” begin
do replace “__I_AM_THE_MAIN__” main_class
do eval { print(“run `mvn package` to create an executable jar”)} [3]
end
with Project p begin
do eval { print(“Find a run script in your project’s bin directory. You’ll have to make it executable yourself, sorry”) }
do addFile “bin/run” runScript
end

Originally published at blog.jessitron.com on January 13, 2017.

It’s Atomist Time!

I’m hella excited to get to work on Atomist full-time starting now (January 2017). Why? What do they do? Oh let me tell you!
I love developing software, not least because we (as an industry) have not yet figured out how to software. We know it’s powerful, but not yet how powerful. Software is like engineering except that the constraints aren’t in what we can build, but what we can design and specify. Atomist is expanding this capacity.
Atomist builds tooling to smooth some bumps in software development. There are three components that I’m excited about, three components that open new options in how we develop software.

Component 1: Code that changes code

First, there are code editors, called Rugs. On the surface, these automate the typing part. Like code generators, except they continue to work with the code after you modify it. Like refactorings in an IDE, except they appear as a pull request, and then you can continue development on that branch. If you have some consistent code structure (and if you use a framework, you do), Rugs can perform common feature-adding or upgrading or refactoring operations. Use standard Rugs to, say, add graph database support to an existing Spring Boot project. Customize Rugs to set up your Travis build uniformly in your projects. Create your own Rugs to implement metrics integration according to your company’s standards — and to upgrade existing code when those standards change.
On the surface this is an incremental improvement over existing code generation and IDE refactoring tools. Yet, I see it as something more. I see it as a whole new answer to the question of “indirection or repetition?” in code. Take for instance: adding a field to a Rails app makes us change the controller, the model, and four other places. Or creating a new service means changing deployment configuration, provisioning, and service discovery. Whenever a single conceptual change requires code changes in multiple spots, we complain about the work and we make mistakes. Then we start to get clever with it: we introduce some form of indirection that localizes that change to one place. Configuration files get generated in the build, Ruby metaprogramming introduces syntax that I can’t even figure out how it’s executable — magic happens. The code gets less explicit, so that we can enforce consistency and make changing it … well, I’m not gonna say “easier” because learning to cast the spell is tricky, but it is less typing.
Atomist introduces a third alternative: express that single intention (“create a new service” or “add this field”) as a Rug editor. This makes writing it one step, and then the editor makes all those code changes in a single commit in a branch. From there, customize your field or your new service; each commit that you make shows how your feature is special. The code remains explicit, without additional magic. When I come back and read it, I have some hope of understanding what it’s doing. When I realize that I forgot something (“oops! I also need to add that service to the list of log sources”) then I fix it once, in the NewService.rug editor. Now I never forget, and I never have to remember.
I love this about developing with Rugs: as I code, I’m asking myself, “how could I automate this?” and then what I learn is encoded in the Rug, for the benefit of future-me and (if I publish it) of future-everyone-else. That is when I feel productive.

Component 2: Coordination between projects

Editors are cute when applied to one project. When applied across an organization, they start to look seriously useful. Imagine: A library released a security update, and we need to upgrade it across the organization. Atomist creates a pull request on every project that uses that library. The build runs, maybe we even auto-merge it when the build passes. Or perhaps there are breaking changes; the editor can sometimes be taught how to make those changes in our code.
And if a Rug can change the way we use a library, then it can change the way we use ours. This is cross-repository refactoring: I publish an internal library, and I want to rename this function in the next version. Here’s my game: I publish not only the new version of my library, but an editor – and then I ask Atomist to create pull requests across the organization. Now it is a quick code review and “accept” for teams to upgrade to the new version.
Atomist coordinates with teams in GitHub and in Slack. Ask Atomist in Slack to start that new feature for you, or to check all repositories in the organization and create pull requests. Atomist can also coordinate with continuous integration. It ties these pieces together across repositories, and including humans. It can react to issues, to build results, to merges; and it can ping you in Slack if it needs more information to act appropriately. I have plans to use this functionality to link libraries to the services that use them: when the build passes on my branch, go build the app that uses my library with this new version, and tell me whether those tests pass.
This is cross-repository refactoring and cross-repository build coordination. This gives companies an alternative to the monorepo, to loading all their libraries and services into one giant repository in order to test them together. The monorepo is a lie: our deployments are heterogenous, so while the monorepo is like “look at this lovely snapshot of a bunch of code that works together” the production environment is something different. The monorepo is also painful because git gets slow when the repository gets large; because it’s hard to tell which commits affect which deployed units; and because application owners lose control over when library upgrades are integrated. Atomist will provide a layer on top of many repositories, letting us coordinate change while our repositories reflect production realities.
Atomist tooling will make multirepo development grow with our codebases.

Component 3: is still a secret

I’m not sure I can talk about the third piece of possibility-expanding tooling yet. So have this instead:
Automated coordination among systems and people who interact with code — this is useful everywhere, but it’s a lot of work to create our own bots for this. Some companies put the resources into creating enough automation for their own needs. No one business-software-building organization has a reason to develop, refine, and publish a general solution for this kind of development-process automation. Atomist does.
When it becomes easy for any developer to script this coordination and the reactions just happen — “Tell me when an issue I reported was closed” “Create a new issue for this commit and then mark it closed as soon as this branch is merged” — then we can all find breakages earlier and we can all keep good records. This automates my work at a higher level than coding. This way whenever I feel annoyed by writing a status report, or when I forget to update the version in one place to match the version in another, my job is not to add an item to a checklist. My job is to create an Atomist handler script to make that happen with no attention from me.

My secret

I love shaving yaks. Shaving them deeply, tenderly, finding the hidden wisdom under their hair. I love adding a useful feature, and then asking “How could that be easier?” and then “How could making that easier be easier?” This is Atomist’s level of meta: We are making software to make it easier for you to make your work easier, as you work to make software to make your customers’ lives easier.
I think we’re doing this in depths and ways other development tools don’t approach. At this level of meta (software for building software for building software for doing work), there’s a lot of leverage, a lot of potential. This level of meta is where orders-of-magnitude changes happen. Software changes the world. I want to be part of changing the software world again, so we can change the real world even faster.
With Atomist, I get to design and specify my own reality, the reality of my team’s work. (Atomist does the operations bit.) Without spending tons of time on it! Well, I get to spend tons of time on it because I get to work for Atomist, because that’s my thing. But you don’t have to spend tons of time on it! You get to specify what you want to happen, in the simplest language we can devise.
We’re looking for teams to work with us on alpha-testing, if you’re interested now. (join our slack, or email me) Let’s learn together the next level of productivity and focus in software development.