Design by implementation

We can develop things faster than we can define them.


I’ve remarked before that @cdupuis can implement new features way faster than I can document them. This is not ridiculous; creating something is often faster than connecting it to existing people who can use it.

Ryan took this farther: we can’t even define what we want faster than we can implement it, sometimes. This is not ridiculous; design informs implementation, and implementation informs design.

Implementation informs design, and the process of implementation builds a mental model in the developer’s head. Making that model concrete and communicating it to someone else is hard, sometimes harder than getting the code working. Communicating that model in a way that oodles of people can use it is harder still, sometimes an order of magnitude harder. It includes documentation, yes, and marketing and customer service and advocacy. If you’re really serious, writing a specification. These all inform the next iterations of design and of implementation.

Developing things is only a beginning.


Developers have a love-hate relationship with code re-use. As in, we used to love it. We love our code and we want it to run everywhere and help everyone. We want to get faster with time by harnessing the work of our former selves.
And yet, we come to hate it. Reuse means dependencies. It means couplings. It means surprises, when changing code impacts something we did not expect, or else it means don’t touch it, it’s too scary. It means trusting code we don’t understand because it’s code didn’t write.

Here’s the thing: sharing code is dangerous. Do it sparingly.

When reuse is bad

Let’s talk about sharing code. Take a business, developing software for its employees or its customers. Let’s talk about code within an organization that is referenced in more than one service, or by multiple flows in a monolith. (Monolith is defined as “one deployable unit maintained by more than one small team.”)

Let’s see some pictures. Purple Service here has some classes or functions that it finds useful, and the team thinks these would be useful elsewhere. Purple team breaks this code out into a library, the peachy circle.

purple circle, peach circle inside

Then someone from Purple team joins Blue team, and uses that library in Blue Service. You think it looks like this:

peach circle under blue and purple circles

Nah, it’s really more like this:

purple circle with peach circle inside. Blue circle has a line to peach circle

This is called coupling. When Purple team changes their library, Blue team is affected. (If it’s a monolith, their code changed underneath them. I hope they have good tests.)
Now, you could say, Blue team doesn’t have to update their version. The level of reuse is the release, we broke out the library, so this is fine.

picture of purple with orange circle, blue with peach circle.

At that point you’ve basically forked, the code isn’t shared anymore. When Blue team needs to make their own changes, they first must upgrade, so they get surprised some unpredictable time later. (This happened to us at Outpace all the time with our shared “util” libraries and it was the worst. So painful. Those “timesavers” cost us a lot of time and frustration.)

This shared code is a coupling between two services that otherwise have nothing to do with each other. The whole point of microservices was to decouple! To make it so our changes impact only code that our team operates! dead. and for what?

To answer that, consider the nature of the shared code. Why is it shared?
Perhaps it is unrelated to the business: it is general utilities that would otherwise be duplicated, but we’re being DRY and avoiding the extra work of writing and testing and debugging them a second time. In this case, I propose: cut and paste. Or fork. Or best of all, try a more formalized reuse-without-sharing procedure [link to my next post].

What if this is business-related code? What if we had good reason to DRY it out, because it would be wrong for this code to be different in Purple Service and Blue Service? Well sorry, it’s gonna be different. Purple and Blue do not have the same deployment schedules, that’s the point of decoupling into services. In this case, either you’ve made yourself a distributed monolith (requiring coordinated deployments), or you’re ignoring reality. If the business requires exactly one version of this code, then make it its own service.

picture with yellow, purple, and blue circles separate, dotty lines from yellow to purple and to blue.

Now you’re not sharing code anymore. You’re sharing a service. Changes to Peachy can impact Purple and Blue at the same time, because that’s inherent in this must-be-consistent business logic.

It’s easier with a monolith; that shared code stays consistent in production, because there is only one deployment. Any surprises happen immediately, hopefully in testing. In a monolith, if Peachy is utility classes or functions, and Purple (or Blue) team wants to change them, the safest strategy is: make a copy, use the copy, and change that copy. Over time, this results in less shared code.

This crucial observation is #2 in Modern Software Over-engineering Mistakes by RMX.

“Shared logic and abstractions tend to stabilise over time in natural systems. They either stay flat or relatively go down as functionality gets broader.”

Business software is an expanding problem. It will always grow, and not with more of the same: it will grow in ways you didn’t plan for. This kind of code must optimize for change. Reuse is the enemy of change. (I’m talking about reuse of internal code.)

Back in the beginning, Blue team reused the peach library and saved time writing code. But writing code isn’t the expensive part, compared to changing code. We don’t add features faster as our systems get larger and we have more code hypothetically available for re-use. We add features more slowly, because every change has more impacts and is less safe. Shared code makes change less safe. The only code safe to share is code that doesn’t change. Which means no versioning. Heck, you might as well have cut and pasted it.

When reuse is good

We didn’t advance as an industry by rewriting, or cut and pasting, everything we need over and over. We build on libraries published by developers and companies all over the globe. They release them, we reuse them. Yes, we get into dependency hell, but it beats writing your own web framework. We get reuse not only of the code, but of understanding: Rails knowledge transfers between employers.

There is a tipping point where reuse is magical.

I argue that this point is well past a release, past a separate jar.
It is past a stable API
past a coherent abstraction
past automated tests
past solid documentation…

All these might be achieved within the organization if responsibility for the shared utilities lives in a separate team; you can try to use Conway’s Law to enforce architectural boundaries, but within an org, those boundaries are soft. And this code isn’t your business, and you don’t have incentives to spend the time on these. Why have backwards compatibility when you can perform human coordination instead? It isn’t worth it. In my past organizations, shared code has instead been the responsibility of no one. What starts out as “leverage” becomes baggage, as all the Ruby code is tied to an old version of Sinatra. Some switch to Go to get a clean slate.
Break those chains! Copy the pieces you need out of that internal library and make them yours.

At the level of winning reuse, that code has its own marketing department
its own sales team
its own office manager
its own stock price.

The level of reuse is the company.

(Pay for software.)

When the responsible organization succeeds by making its code stable and backwards-compatible and easy to work with and well-documented and extensively tested, that is code I want to reuse!

In addition to SaaS companies and vendors, there are organizations built around open-source software. This is why we look for packages and frameworks with a broad community around them. Or better, a foundation for keeping shared projects healthy. (Contribute to them.)


Reuse is dangerous because it introduces coupling. Share business code only when that coupling is inherent to the business domain. Share library and utility code only when it is maintained by an organization dedicated to publishing that code. (Same with services. If you can pay for infrastructure-level tools, you’ll get better tools without distracting your organization.)

Why did we want to reuse internal code anyway?
For speed, but speed of change is more important.
For consistency, but that means coupling. Don’t hold your teams back with it.
For propagation of bug fixes, which I’ve not seen happen.

All three of these can be automated [LINK to my next post] without dependencies.

Next time you consider making your code reusable, ask “who will I sell this to?”
Next time someone (including you) suggests you reuse their code, ask “who publishes that?” and if they say “me,” copy it instead.


How important is correctness?

This is a raging debate in our industry today. I think the answer depends strongly on the kind of problem a developer is trying to solve: is the problem contracting or expanding? A contracting problem is well-defined, or has the potential to be well-defined with enough rigorous thought. An expanding problem cannot; as soon as you’ve defined “correct,” you’re wrong, because the context has changed.

A contracting problem: the more you think about it, the clearer it becomes. This includes anything you can define with math, or a stable specification: image conversion, what do you call it when you make files smaller for storage. There are others: ones we’ve solved so many times or used so many ways that they stabilize: web servers, grep. The problem space is inherently specified, or it has become well-defined over time.
Correctness is possible here, because there is such a thing as “correct.” Programs are useful to many people, so correctness is worth effort. Use of such a program or library is freeing, it scales up the capacity of the industry as a whole, as this becomes something we don’t have to think about.

An expanding problem: the more you think about it, the more ways it can go. This includes pretty much all business software; we want our businesses to grow, so we want our software to do more and different things with time. It includes almost all software that interacts directly with humans. People change, culture changes, expectations get higher. I want my software to drive change in people, so it will need to change with us.
There is no complete specification here. No amount of thought and care can get this software perfect. It needs to be good enough, it needs to be safe enough, and it needs to be amenable to change. It needs to give us the chance to learn what the next definition of “good” might be.

I propose we change our aim for correctness to an aim for safety. Safety means, nothing terrible happens (for your business’s definition of terrible). Correctness is an extreme form of safety. Performance is a component of safety. Security is part of safety.

Tests don’t provide correctness, yet they do provide safety. They tell us that certain things aren’t broken yet. Process boundaries provide safety. Error handling, monitoring, everything we do to compensate for the inherent uncertainty of running software in production, all of these help enforce safety constraints.

In an expanding software system, business matters (like profit) determine what is “good enough” in an expanding system. Risk tolerance goes into what is “safe enough.” Optimizing for the future means optimizing our ability to change.

In a contracting solution, we can progress through degrees of safety toward correctness, optimal performance. Break out the formal specification, write great documentation.

Any piece of our expanding system that we can break out into a contracting problem space, win. We can solve it with rigor, even make it eligible for reuse.

For the rest of it – embrace uncertainty, keep the important parts working, and make the code readable so we can change it. In an expanding system, where tests are limited and limiting, documentation becomes more wrong every day, the code is the specification. Aim for change.

Philly ETE, Design in the small

Every conference has little themes for me. Here’s one from Philly ETE this week.

The opening keynote from Tom Igoe, founder of Arduino, displayed examples of unique and original projects. Many of them were built for one person – a device to tell her mother whether she’s taken this pill already today, or to help his father play guitar again after a stroke. Others were created for one place: this staircase becomes a piano. Arduino enables design in the small: design completely customized for a context and an audience. A product that meets these local needs, that works; no other requirements.

Diana Larsen does something similar for agile retrospectives. She pointed out that repeating the same retrospective format shows diminishing returns. Before a 90-minute retro, Diana spends 1-3 hours preparing: gathering data, choosing a theme and an activity. She has high-scale models of how people learn, yet each learning opportunity is custom-designed for the context and people. This has great returns: when a team gains the skill of thoughtful retrospectives, all team meetings are transformed.

Individualization appeared in smaller ways on the second day: Erik mentioned the compiler flags available in the Typelevel Scala compiler, allowing people to choose their language features. Aaron Bedra customized security features to the attack at hand: a good defense is intentional, as thoughtfully built as product features. Every one different.

Finally, Rebecca Wirfs-Brock debunked the agile aversion to architecture. Architecture tasks can be scaled and timeboxed based on the size and cruciality of the project. Architecture deliverables meet current needs, work for the team, and nothing else. It’s almost like designing each design process to suit the context.

This is the magic that agile development affords us: suiting each process, each technology, each step to the local context. That helps us do our best work. The tricky bit is keeping that context in mind at all levels. From my talk: the best developers are reasoning at all scales, from the code details of this function to the program to the software system to the business of why are we doing this at all. This helps us do the right work. Maintaining the connection from what we’re doing to why we’re doing it, we can make better decisions. And as Tom said in the closing sentence of the opening keynote, we can remember: “The things we make are less important than the relationships they enable.”

This was one of the themes I took from PhillyETE. Other takeaways were tweeted. There were many great talks, and InfoQ filmed them, so I’ll link them in this post later. This is conference is thoughtfully put together. Recommend.

Application vs. Reference Data

At my very first programming job, I learned a distinction between two kinds of data: Reference and Application.

Application data is produced by the application, consumed and modified by the software. It is customers, orders, events, inventory. It changes and grows all the time. Typically a new deployment of the app starts with no application data, and it grows over time.

Reference data is closer to configuration. Item definitions, tax rates, categories, drop-down list options. This is read by the application, but changed only by administrative interfaces. It’s safe to cache reference data; perhaps it updates daily, or hourly, at the most. Often the application can’t even run without reference data, so populating it is part of deployment.

Back at Amdocs we separated these into different database schemas, so that the software had write access to application data and read-only access to reference data. Application data had foreign key relationships to reference data; inventory items referenced item definitions, customers referenced customer categories. Reference data could not refer to application data. This follows the Stable Dependencies principle: frequently-changing data depended on rarely-changing data, never the other way around.

These days I don’t go to the same lengths to enforce the distinction. It may all go in the same database, there may be no foreign keys or set schemas, but in my head the classification remains. Which data is essential for application startup? Reference. Which data grows and changes frequently? Application. Thinking about this helps me avoid circular dependencies, and keep a clear separation between administration and operation.

My first job included some practices I shudder at now[1], but others stick with me. Consider the difference between Reference and Application the next time you design a storage scheme.

[1] Version control made of perl on top of cvs, with file locking. Unit tests as custom drivers that we threw away. C headers full of #DEFINE. High-level design documents, low-level design documents, approvals, expensive features no one used. Debugging with println… oh wait, debugging with println is awesome again. 

A victory for abstraction, re-use, and small libraries

The other day at Outpace, while breaking some coupling, Eli and I decided to retain some information from one run of our program to another. We need to bookmark how far we read in each input data table. How can we persist this small piece of data?

Let’s put it in a file. Sure, that’ll work.[1] 

Next step, make an abstraction. Each of three configurations needs its own “how to read the bookmark” and “how to write the bookmark.”[2] What can we name it?

After some discussion we notice this is basically a Clojure atom – a “place” to store data that can change – except persistent between runs.

Eli googles “clojure persist atom to disk” and bam! He finds a library. Enduro, by @alandipert. Persistent atoms for Clojure, backed by a file or Postgres. Complete with an in-memory implementation for testing. And thread safety, which we would not have bothered with. Hey, come to think of it, Postgres is a better place to store our bookmarks.

From a need to an abstraction to an existing implementation! with better ideas! win!

Enduro has no commits in the last year, but who cares? When a library is small enough, it reaches feature-completion. For a solid abstraction, there is such a thing as “done.”

Now, it happens that the library isn’t as complete as we hoped. There are no tests for the Postgres implementation. The release! method mentioned in the README doesn’t exist.

But hey, we can add these to the library faster and with less risk than implementing it all ourselves. Alan’s design is better than ours. Building on a solid foundation from an expert is more satisfying that building from scratch. And with pull requests, everybody wins!

This is re-use at its best. We paused to concentrate on abstraction before implementation, and it paid off.

[1] If something happens to the file, our program will require a command-line argument to tell it where to start.

[2] In OO, I’d put that in an object, implementing two single-method interfaces for ISP, since each function is needed in a different part of the program. In Clojure, I’m more inclined to create a pair of functions. Without types, though, it’s hard to see the meaning of the two disparate elements of the pair. The best we come up with is JavaScript-object-style: a map containing :read-fn and :write-fn. At least that gives them names.

Software is a tree

Maybe software is like a tree.

The applications, websites, games that people use are the leaves. They’re the important part of the tree, because they’re useful. They solve real problems and improve lives, they make money, they entertain.

The leaves are built upon the wood of the tree, branches and trunk: all the libraries and languages and operating systems and network infrastructure. Technologies upon technologies, upon which we build applications. They’re the important part of the tree because dozens to thousands of leaves depend on each piece of wood.

Which part of the software tree do you work on? Is it a leaf? Leaves need polished according to their purpose, monitored according to their criticality, grown according to users’ real needs. Is it the wood? The wood needs to be strong, very well-tested because it will be used in ways its authors did not imagine.

Ever find yourself in between, changing an internal library that isn’t tested and documented like a strong open-source library, but isn’t specific to one application either? I’ve struggled with code like this; we want to re-use it but when we write it we’re trying to serve a purpose – grow the leaf – so we don’t stop to write property tests, handle edge cases, make the API clear and flexible. This code falls in the uncanny valley between wood and leaf. I don’t like it.

Code that is reusable but not core to the business function is best shared. Use another library, publish your own, or else stagnate in the uncanny valley. Follow the mainstream, form a new main
stream, or get stuck in a backwater.

With a few days set aside to focus on it, we could move that shared code into the wood, and release it as open source. Make it public, then document, test, and design with care. When we graft our twig into the larger tree, it can take nourishment from the contributing community, and maybe help more leaves sprout.

If the code isn’t useful enough to release as even a tiny open-source library, then it isn’t worth re-using. Build it right into multiple leaves if multiple leaves need it. App code is open for modification, instead of the closed-for-modification-but-needs-modification of the shared internal library.

With care, ours can be a healthy tree: millions of shiny custom leaves attached to strong shared branches.

Evolution of a developer


At first I wrote flexible code. I figured that because I couldn’t predict the future, I better leave all the options open. The code was modularized, well-factored, with a lot of bells and whistles, and more than enough documentation. I was proud of it, until I learned that:

  • Things changed so rapidly that even my solidest assumptions became false. 
  • The small flexibilities in the code were too small to be really helpful, yet too many to be easily refactored.
  • In a world of 140 characters, nobody read my heavy documentation. Literate programming didn’t work, if ever.
  • Longer code, no matter good or bad, is harder to maintain.

So, in another project, I switched to writing extremely tight code. I used a lot of functional abstractions and removed every single bit of redundancy. And documentation is, of course, left as an exercise for the reader. I figured that good code was deleted code, and short code could be easily refactored. I was proud of it, until I learned:

  • 80% of the efforts had been spent on reducing 20% of the redundancy. Like speeding, it didn’t save much. 
  • Highly-factored, redundancy-free code can be difficult to document, understand and debug. 
  • The best way to compress some code is using gzip, not your hand.

Now I think, just like many things in life, a balanced position is the best. In other words: write simple code. I deeply believe that you can achieve simplicity without compromising on other aspects. I believe that applying the simplest solution which consumes minimal energy is how the whole universe works. When you get one simple answer, it may not be the right answer. However, once you find the right answer, it is usually also very simple. 

I think we should just make it easy. If you feel difficulty when writing a program, if you find yourself scratching your head, either because of intricate structural design or abstract code factorization, you are probably doing it wrong. After all, you should be able to enjoy programming, not suffer through it. There is more than one way to do it – any approach can be a good start, as long as you feel easy and comfortable about it, and you do not forget to keep improving it. If it’s too fat, cut some; if it’s too thin, grow some. Eventually you’ll reach a point where you find ultimate simplicity. Call it fixed/optimal/equilibrium point as you like, it’s all the same thing. It’s just how the universe works.

— Ming Lei, on the St. Louis Lambda Lounge list

Don Quixote was an Enterprise Architect

We need to invent a word:

?: (n) The goal that you aim for, not because you expect to hit it, but because aiming for it points you in the right direction.

Julian Browne in The Death of Architecture accepts that while an Enterprise Architecture will never be achieved, it can direct our efforts: “Great ideas and plans are what inspires and guides people to focus on making those tactical changes well.”

Having a plan is good. Sticking religiously to the plan is bad. The business is constantly changing, and so the architecture should align with the current status and the latest goals. Architecture is a map, but we operate in Street View. The map is always out of date.

So it is in life. As people we are constantly changing. There are objectives to shoot for, some concrete and achievable, others unrealistic but worth trying. Aim for enlightenment, if you like, but redefine what enlightenment means with each learning. Embrace change, and direct it by choosing where to focus your attention. If our goal is readable code, we’re never going to transform our entire legacy codebase to our current coding standards. Our coding standards will have changed by then. Instead, as we modify pieces of the code, we make these parts the best they can be according to the direction we’ve set.

Aim for the mountaintop, but recalibrate the location of that top. Appreciate each foot of ascent, as the good-software mountain is constantly shifting, and each improvement — each refactor or new test or simplification — counts as progress. An architecture achieved is an architecture out of date – touch the sky, find that it is made of paper, tear through it and aim again for the highest thing you can see.

Why Functional Matters: Your white board will never be the same

Why learn functional programming? For better design skills!

The other day we designed a process to match cleared deposits with dispensations. This is how I would have white-boarded it a few years ago:

Since then I’ve played around with functional programming. It encourages one to think about processing in terms of data streams: data comes in, gets transformed and filtered and manipulated, and finally results are output. Not only do programs work this way, but each piece of the program — each function — is also modeled as data in, data out. No state changes in the middle.

Thinking about the problem in this way leads to this kind of diagram:

Thinking about the program as a data pipeline keeps me thinking about what needs to happen, instead of how each step should be done. It’s a higher level of thinking that keeps us from bogging down in implementation details at the design phase.

Whatever language we use to write the solution, thinking about it this way has the following advantages:
* It breaks down into pieces. (The orange boxes represent JIRA tasks.) The requirements are already specified on the diagram: what goes in, what comes out. Each task can be developed independently.
* Each bit is testable. The database-impacting pieces are called out; other than that, each box is defined entirely by input and output. That’s the easiest kind of component to test.

In this way, thinking functionally helps with agile (task breakdown), TDD, and maintainability. The code is more composable. Thinking about the problem is easier because we can see what the data looks like at each step.
Independent, testable components: that’s functional.

For how functional thinking helps at a lower level too, check this post.

New coding tricks are fun, but new white board techniques are serious business.