Sunday, February 15, 2015

Microservices, Microbusinesses

To avoid duplication of effort, we can build software uniformly. Everything in one monolithic system or (if that gets unwieldy) in services that follow the same conventions, in the same language, developed with uniform processes, released at the same time, connecting to the same database.

That may avoid duplication, but it doesn't make great software. Excellence in architecture these days involves specialization -- not by task, but by purpose. Microservices divide a software system into fully independent pieces. Instead of limiting which pieces talk to which, they specify clear APIs and let both the interactions and the innards vary as needed.

A good agile team is like a microservice: centered around a common purpose. In each retro, the team looks for ways to optimize their workings, ways particular to that team's members, context, objectives.

When a service has a single purpose[1], it can focus on the problems important to that purpose. Authentication? availability is important, consistency, and security. Search? speed is crucial, repeatability is not. Each microservice can use the database suited to its priorities, and change it out when growth exceeds capacity. The business logic at the core of each service is optimized for efficient and clear execution.

Independence has a cost. Each service is a citizen in the ecosystem. That means accepting requests that come in, with backwards compatibility. It means sending requests and output in the format other services require, not overloading services it depends on, and handling failure of any downstream service. Basically, everybody is a responsible adult.
That's a lot of overhead and glue code. Every service has to do translation from input to its internal format, and then to whatever output format someone else requires. Error handling, caching or throttling, failover and load balancing and monitoring, contract testing, maintaining multiple interface versions, database interaction details. Most of the code is glue, layers of glue protecting a small core of business logic. These strong boundaries allow healthy relationships with other services, including new interactions that weren't designed into the architecture from the beginning. For all this work, we get freedom on the inside. We get the opportunity to exceed expectations, rather than straining standardized languages and tools to meet requirements.

Do the teams in your company have the opportunity to optimize in the same way?

I've worked multiple places where management decreed that all teams would track work in JIRA, with umpteen required fields, because they like how progress tracking rolls up. They can see high-level numbers or drill down into each team's work. This is great for legibility.[3] All the work fits into nice little boxes that add together. However, what suits the organizational hierarchy might not suit the work of an individual team.

Managers love to have a standard programming language, standard testing tools with reports that roll up, standard practices. This gives them the feeling that they can move people from team to team with less impact to performance. If that feeling is accurate, it's only because the level of performance is constrained everywhere.

Like software components, teams have their own problems to solve. Communication between individuals matters most, so optimize for that[2]. Given the freedom to vary practices and tools, a healthy agile team gets better and better. The outcome of a given day's work is not only a task completed: it includes ideas about how to do this better next time.

Tom Marsh says, "Make everyone in your organisation believe that they are working in a business unit about 10 people big." A team can learn from decisions and make better ones and spiral upward into exceptional performance (plus innovation), when internal consensus is enough to implement a change. Like a microbusiness.

Still, a team exists as a citizen inside a larger organization. There are interfaces to fulfill. Management really does need to know about progress. Outward collaboration is essential. We can do this the same way our code does: with glue. Glue made of people. One team member, taking the responsibility of translating cards on the wall into JIRA, can free the team to optimize communication while filling management's strongest needs.

Management defines an API. Encapsulate the inner workings of the team, and expose an interface that makes sense outside. By all means, provide a reference implementation: "Other teams use Target Process to track work." Have default recommendations: "We use Clojure here unless there's some better solution. SQL Server is our first choice database for these reasons." Give teams a strong process and technology stack to start from, and let them innovate from there.

On a healthy team, people are accomplishing something together, and that's motivating. When we feel agency to share and try out ideas, when the external organization is only encouraging, then a team can gel, excel and innovate. This cohesive team culture (plus pairing) brings new members up to speed faster than any familiarity with tooling.

As in microservices, it takes work to fulfill external obligations and allow internal optimization. This duplication is not waste. Sometimes a little overhead unleashes all the potential.


[1] The "micro" in microservices is a misnomer: the only size restriction on a microservice is Conway's Law: each is maintained by a single team, 10 or fewer people. A team may maintain one or more system components, and one team takes full responsibility for the functioning of the piece of software.

[2] Teams work best when each member connects with each other member (NYTimes)

[3] Seeing Like a State points out how legibility benefits the people in charge, usually at the detriment of the people on the ground. Sacrificing local utility for upward legibility is ... well, it's efficiency over innovation.

Tuesday, February 10, 2015

Fun with Optional Typing: cheap mocking


For unit tests, it's handy to mock out side-effecting functions so they don't slow down tests.[1] Clojure has an easy way to do this: use with-redefs to override function definitions, and then any code within the with-redefs block uses those definitions instead.

To verify the input of the side-effecting function, I can override it with something that throws an exception if the input is wrong.[2] A quick way to do that is to check the input against a schema.

That turns out to be kinda pretty. For instance, if I need to override this function fetch-orders, I can enforce that it receives exactly the starting-date I expect, and a second argument that is not specified precisely, but still meets a certain condition.

(with-redefs [fetch-orders (s/fn [s :- (s/eq starting-date)
                                  e :- AtLeastAnHourAgo]
                            [order])]
... )

Here, the s/fn macro creates a function that (when validation is activated[3]) checks its input against the schemas specified after the bird-face operator. The "equals" schema-creating function is built-in; the other I created myself with a descriptive name. The overriding function is declarative, no conditionals or explicit throwing or saving mutable state for later.

If I have a bug that switches the order of the inputs, this test fails. The exception that comes out isn't pretty.
expected: (= expected-result (apply function-under-test input))
  actual: clojure.lang.ExceptionInfo: Input to fn3181 does not match schema: [(named (not (= #<DateTime 2014-12-31T23:55:00.000Z> a-org.joda.time.DateTime)) s) nil]
Schema isn't there yet on pretty errors. But hey, my test reads cleanly, it was simple to write, and I didn't bring in a mocking framework.

See the full code (in the literate-test sort of style I'm experimenting with) on github.




[1] for the record, I much prefer writing code that's a pipeline, so that I only have to unit-test data-in, data-out functions. Then side-effecting functions are only tested in integration tests, not mocked at all. But this was someone else's code I was adding tests around.

[2] Another way to check the output is to have the override put its input into an atom, then check what happened during the assertion portion of the test. Sometimes that is cleaner.

[3] Don't forget to (use-fixtures :once schema.test/validate-schemas) 

Saturday, January 31, 2015

Cropping a bunch of pictures to the same dimensions

Ah, command line tools, they're so fast. And so easy to use on a Mac.

Given a bunch of image files in the same dimensions, that you want to crop to a fixed portion of the image:

1) Install imagemagick
brew install imagemagick
2) put all the images in a directory by themselves, and cd to that directory in the terminal

3) check the size of one of them using an imagemagick command-line utility:

identify IMG_1400.jpg
IMG_1400.jpg JPEG 960x1280 960x1280+0+0 8-bit sRGB 434KB 0.000u 0:00.000

Oh look, that one has a width of 960 and a height of 1280.

4) crop one of them, look at it, tweak the numbers, repeat until you get the dimensions right:
convert IMG_1400.jpg -crop 750x590+60+320 +repage test.jpg
Convert takes an input file, some processing instructions, and an output file. Here, I'm telling it to crop the image to this geometry (widthxheight+xoffset+yoffset), and then make the output size match what we just cropped it to.

The geometry works like this: move down by the y offset and to the right by the x offset. From this point, keep the portion below and to the right that is as wide as width and as tall as height.

5) Create an output directory.
mkdir output
6) Figure out how to list all your input files. Mine are all named IMG_xxxx.jpg so I can list them like this:
ls IMG_*.jpgIMG_1375.jpg IMG_1380.jpg IMG_1385.jpg...
7) Tell bash to process them all:[1]
for file in `ls IMG*.jpg`
do
echo $file
convert $file  -crop
750x590+60+320 +repage output/$file
done
8) Find the results in your output directory, with the same names as the originals.

-----
[1] in one line:
for file in `ls IMG*.jpg`;> do echo $file; convert $file  -crop 7750x590+60+320 +repage out/$file; done

Tuesday, January 20, 2015

Application vs. Reference Data

At my very first programming job, I learned a distinction between two kinds of data: Reference and Application.

Application data is produced by the application, consumed and modified by the software. It is customers, orders, events, inventory. It changes and grows all the time. Typically a new deployment of the app starts with no application data, and it grows over time.

Reference data is closer to configuration. Item definitions, tax rates, categories, drop-down list options. This is read by the application, but changed only by administrative interfaces. It's safe to cache reference data; perhaps it updates daily, or hourly, at the most. Often the application can't even run without reference data, so populating it is part of deployment.


Back at Amdocs we separated these into different database schemas, so that the software had write access to application data and read-only access to reference data. Application data had foreign key relationships to reference data; inventory items referenced item definitions, customers referenced customer categories. Reference data could not refer to application data. This follows the Stable Dependencies principle: frequently-changing data depended on rarely-changing data, never the other way around.

These days I don't go to the same lengths to enforce the distinction. It may all go in the same database, there may be no foreign keys or set schemas, but in my head the classification remains. Which data is essential for application startup? Reference. Which data grows and changes frequently? Application. Thinking about this helps me avoid circular dependencies, and keep a clear separation between administration and operation.

My first job included some practices I shudder at now[1], but others stick with me. Consider the difference between Reference and Application the next time you design a storage scheme.


[1] Version control made of perl on top of cvs, with file locking. Unit tests as custom drivers that we threw away. C headers full of #DEFINE. High-level design documents, low-level design documents, approvals, expensive features no one used. Debugging with println... oh wait, debugging with println is awesome again. 

Saturday, January 17, 2015

Fun with Optional Typing: narrowing errors

After moving from Scala to Clojure, I miss the types. Lately I've been playing with Prismatic Schema, a sort of optional typing mechanism for Clojure. It has some surprising benefits, even over Scala's typing sometimes. I plan some posts about interesting ones of those, but first a more ordinary use of types: locating errors.

Today I got an error in a test, and struggled to figure it out. It looked like this:[1]

expected: (= [expected-conversion] result)
  actual: (not (= [{:click {:who {:uuid "aeiou"}, :when #<DateTime 2014-12-31T23:00:00.000Z>}, :outcome {:who {:uuid "aeiou"}, :when #<DateTime 2015-01-01T00:00:00.000Z>, :what "bought 3 things"}}] ([{:click {:who {:uuid "aeiou"}, :when #<DateTime 2014-12-31T23:00:00.000Z>}, :outcome {:who {:uuid "aeiou"}, :when #<DateTime 2015-01-01T00:00:00.000Z>, :what "bought 3 things"}}])))

Hideous, right? It's super hard to see what's different between the expected and actual there. (The colors help, but the terminal doesn't give me those.)

It's hard to find the difference because the difference isn't content: it's type. I expected a vector of a map, and got a list of a vector of a map. Joy.

I went back and added a few schemas to my functions, and the error changed to

  actual: clojure.lang.ExceptionInfo: Output of calculate-conversions-since does not match schema: [(not (map? a-clojure.lang.PersistentVector))]

This says my function output was a vector of a vector instead of a map. (This is one of Schema's more readable error messages.)

Turns out (concat (something that returns a vector)) doesn't do much; I needed to (apply concat to-the-vector).[2]

Clojure lets me keep the types in my head for as long as I want. Schema lets me write them down when they start to get out of hand, and uses them to narrow down where an error is. Even after I spotted the extra layer of sequence in my output, it could have been in a few places. Adding schemas pointed me directly to the function that wasn't doing what I expected.

The real point of types is that they clarify my thinking and document it at the same time. They are a skeleton for my program. I like Clojure+Schema because it lets me start with a flexible pile of clay, and add bones as they're needed.

-----
[1] It would be less ugly if humane-test-output were activated, but I'm having technical difficulties with that at the moment.
[2] here's the commit with the schemas and the fix.

Monday, January 12, 2015

Readable, or reason-aboutable?

My coworker Tom finds Ruby unreadable.
What?? I'm thinking. Ruby can be quite expressive, even beautiful.
But Tom can't be sure what Ruby is going to do. Some imported code could be modifying methods on built-in classes. You can never be sure exactly what will happen when this Ruby code executes.

He's right about that. "Readable" isn't the word I'd use though: Ruby isn't "reason-aboutable." You can't be completely sure what it's going to do without running it. (No wonder Rubyists are such good testers.)

Tom agreed that Ruby could be good at expressing the intent of the programmer. This is a different goal from knowing exactly how it will execute.

Stricter languages are easier to reason about. In Java I can read the specification and make inferences about what will happen when I use the built-in libraries. In Java, I hate the idea of bytecode modification because it interferes with that reasoning.

With imperative code in Java or Python, where what you see it what you get, you can try to reason about these by playing compiler. Step through what the computer is supposed to do at each instruction. This is easier when data is immutable, because then you can trace back to the one place it could possibly be set.

Beyond immutability, the best languages and libraries offer more shortcuts to reasoning. Shortcuts let you be sure about some things without playing compiler through every possible scenario. Strong typing helps with this: I can be sure about the structure of what the function returns, because the compiler enforces it for me.

Shortcuts are like, I can tell 810 is divisible by 3 because its digits add to a number divisible by 3. I don't have to do the division. This is not cheating, because this is not coincidence; someone has proven this property mathematically.

Haskell is the most reason-aboutable language, because you can be sure that the environment won't affect execution of the code, and vice-versa, outside of the IO monad. Mathematical types like monoids and monads help too, because they come with properties that have been proven mathematically. Better than promises in the documentation. More scalable than playing compiler all day.[1]

"Readability" means a lot of things to different people. For Tom, it's predictability: can he be sure what this code will do? For many, it's familiarity: can they tell at a blink what this code will do? For me, it's mostly intent: can I tell what we want this code to do?

Haskellytes find Haskell the most expressive language, because it speaks to them. Most people find it cryptic, with its terse symbols. Ruby is well-regarded for expressiveness, especially in rich DSLs like RSpec.

Is expressiveness (of the intent of the programmer) in conflict with reasoning (about program execution)?


[1] "How do you really feel, Jess?"

We want to keep our programs simple, and avoid unnecessary complexity. The definition of a complex system is: the fastest way to find out what will happen is to run it. This means Ruby is inviting complexity, compared to Haskell. Functional programmers aim for reason-aboutable code, using all the shortcuts (proven properties) to scale up our thinking, to fit more in our head. Ruby programmers trust inferences made from example tests. This is easier on the brain, both to write and read, for most people. It is not objectively simpler.

Friday, January 9, 2015

Spring cleaning of git branches

It's time to clean out some old branches from the team's git repository. In memory of them, I record useful tricks here.

First, Sharon's post talks about finding branches that are ripe for deletion, by detecting branches already merged. This post covers those, plus how to find out more about the others. This post is concerned with removing unused branches from origin, not locally.

Here's a useful hint: start with
git fetch -p
to update your local repository with what's in origin, including noticing which branches have been deleted from origin.
Also, don't forget to
git checkout mastergit merge --ff-only
so that you'll be on the master branch, up-to-date with origin (and won't accidentally create a merge commit if you have local changes).

Next, to find branches already merged to master:

git branch -a --merged

This lists branches, including remote branches (the ones on origin), but only ones already merged to the current branch. Note that the argument order is important; the reverse gives a silly error.  Here's a one-liner that lists them:
git branch -a --merged | grep -v -e HEAD -e master | grep origin | cut -d '/' -f 3- 
This says, find branches already merged; exclude any references to master and HEAD; include only ones from origin (hopefully); cut out the /remotes/origin/ prefix.

The listed branches are safe to delete. If you're brave, delete them permanently from origin by adding this to the previous command:
 | xargs git push --delete origin
This says, take all those words and put them at the end of this other command, which says "delete these references on the origin repository."

OK, those were the easy ones. What about all the branches that haven't been merged? Who created those things anyway, and how old are they?

git log --date=iso --pretty=format:"%an %ad %d" -1 --decorate

is a lovely command that lists the author, date in ISO format (which is good for sorting), and branches and tags of the last commit (on the current branch, by default).

Use it on all the branches on origin:
git branch -a | grep origin | grep -v HEAD | xargs -n 1 git log --date=iso --pretty=format:"%an %ad %d%n" -1 --decorate | grep -v master | sort
List remote branches; only the ones from origin; exclude the HEAD, we don't care and that line is formatted oddly; send each one through the handy description; exclude master; sort (by name then date, since that's the information at the beginning of the line).

This gives me a bunch of lines that look like:

Shashy 2014-08-15 11:07:37 -0400  (origin/faster-upsert)
Shashy 2014-10-23 22:11:40 -0400  (origin/fix_planners)
Shashy 2014-11-30 06:50:57 -0500  (origin/remote-upsert)
Tanya 2014-10-24 11:35:02 -0500  (origin/tanya_jess/newrelic)
Tanya 2014-11-13 10:04:48 -0600  (origin/kafka)
Yves Dorfsman 2014-04-24 14:43:04 -0600  (origin/data_service)
clinton 2014-07-31 16:26:37 -0600  (origin/warrifying)
clinton 2014-09-15 13:29:14 -0600  (origin/tomcat-treats)

Now I am equipped to email those people and ask them to please delete their stray branches, or give me permission to delete them.