The power of embedded developers

Meet my friend Sean.

Sean with a big smile

Sean is one of the most powerful developers I know. Not best, not brilliantest, but most powerful because: he provides more business value than some 50-person teams.

What is the job title of such a powerful developer? It isn’t “ninja” (although maybe it should be). It’s “Accounting Manager.” WAT.

Program: runs on the author’s machine when the author runs it.
Software: runs on any machine at any time, serving many people’s purposes.

Here’s the thing: Sean can code — he was a top developer when we worked together fifteen years ago — but he no longer writes software. He writes programs instead.

Dual wielding

When it comes to business-value power, domain knowledge adds potential; technical knowledge adds potential; and the combination is drastically better than each.

business knowledge, short bar. Technical knowledge, short bar. Both, very tall bar. Y axis, Value.

Why? In Sean’s words, “because you can actually question what you’re doing.” While breaking a process down in order to encode it as a program, his knowledge of accounting and the business needs lets him distinguish between what’s important or not, and how relevant a particular case is.

Typically, when software impacts a business process flow, it’s because the designers and developers didn’t understand. It’s a negative impact. When the designer/developer is on the business team, process changes emerge from the program design. Codifying well-understood activities reveals opportunities for synergy between businesspeople and the program. It’s a positive impact.

Sean can spot problems, observe trends, and improve processes. Software created from static requirements can only ossify processes.

How he does it

When Sean does reporting, that means: finding out what needs done and why, learning all the databases that have the information and acquiring it, joining millions of rows and performing whatever accounting business logic, putting this into a useful Excel spreadsheet, and distributing that to people – in a couple hours per month.

Those few hours of “doing reporting” each month are possible because the rest of his time is spent thinking, investigating, analyzing, troubleshooting, and programming. Given this freedom, Sean adds more business value every month.

Gardening: gradual improvement of a program. Adding error handling where errors occur, visibility where it’s needed, performance improvements where it counts. Gardening addresses the pain points after they’re felt.

When Sean does reporting, that means: automating every repetitive step, scheduling his programs, monitoring them, and gardening them. What he already does becomes trivial, and then he’s free to do more and more useful work. He and his programs are not separable; they’re both part of getting the job done. And when that job changes, he’s ready.

a person and a box inside a wall

This is an example of the code+coder ecosystem. Sean’s programs aren’t useful without him around to run them and change them, and Sean wouldn’t get much done without the stockpile of code he’s written over the years. The benefit of this system is: flexibility! When the real accountants need more or different information, Sean makes a change in 5 minutes to 3 days.  Compare this to weeks when a small feature is requested of the IT department or software vendor.

There are good reasons the software vendor takes so long. Software should be bulletproof. It’s backed by fixed, negotiated, contracted requirements. It has security, authorization, failover, automated testing. Coordination, contracts, APIs. The difficulty multiplies, and software is 1000x more work to write than a program.

software system: several boxes inside many walls, and lots of people, all inside another wall

Sean doesn’t need all that, because he is the monitoring, the security, the failover. He tests with his eyeballs and his business knowledge as the results appear. As a single point of responsibility, every problem that Sean hits is a learning opportunity, and he has the ability and desire to implement the next program, or the next change, better.

The advantage of software over programs is: stability. The disadvantage is: stability. When a particular program is broadly useful and has hardly changed since Sean implemented it, that’s a good candidate to transfer over IT as a feature request.

Who can do this?

A lot of people can be this powerful. Sean is a developer who learned the business. I know other business-types who have learned to program. It’s wicked effective when it happens. It takes free time, a lot of learning, great management, and a contact network for support.

Having started as a developer with the major software vendor, Sean can talk to DBAs and sysadmins and programmers. His management had to push to get him access to some of these people, and now those communication channels are super valuable both ways. When there’s a problem, the DBAs call him to find out about the significance of various data.

the code-coder ecosystem, underneath an umbrella with a person in it to represent the managerSean’s management values his work, values him, and they work to cut through the red tape. Accounting Managers don’t usually have access to unix machines, production Oracle databases, and automated emailers. Clever people do clever things, and sometimes that surprises the corporate security departments.

The breadth of knowledge, through the business domain to finance to programs to databases to the specific software his programs interact with, doesn’t come for free. And it doesn’t stay for free. There are an infinite number of tools that might help; one person can’t know them all, but it helps to know what’s possible. When Sean took over some other person’s Access databases, it didn’t take him long to knock them down from several gigs to 250MB, because he knew what was possible. Knowing the questions to ask is more important than knowing the answers.

This knowledge doesn’t stay current for free. Most of Sean’s time is spent in meetings, keeping up on what is changing in the many systems he and his programs interact with. This is the limiting factor in growing his power.

One thing you can’t do to people like Sean: budget their time. When a person is production support and development and maintenance and design and interface to many departments, their time is unpredictable. Sometimes Sean will implement a last minute change to compensate for a bug introduced in some other software that can’t be fixed for a few weeks, and he’ll work crazy hours for a couple days. Other times, he can chip away at niggling errors, gardening his programs so that bugs like that will be caught earlier or easier to fix in the future. It isn’t predictable. Sean is flexible, and his management recognizes this. Slack gives us freedom to get better at our jobs.

What can stop us?

Motivation comes for free when people feel valued, but it’s easy to kill. One is cramming our time so full that we don’t have time to improve how we work. Another is red tape: jumping through hoops to get access to data, to get programs installed, to get questions answered. These say, “your ideas are not valued, not shut up and do the same thing every day for the rest of your life.”

There’s HR rules, which say Sean can’t get promoted or get a raise because Finance doesn’t have a job title for “Powerful Accountant-Programmer Combination.”

There’s budgets: Sean could double his power if he had a person to train. Someone to split the meetings with. And someone to carry on his legacy when he moves on to another amazing thing. Right now he’s too essential to move, and that’s another de-motivation.

The Big Problem

And when Sean does move on? In his case, the code-coder ecosystem would collapse. No one else knows his code, so they can’t re-use it. Maybe they could run it (maybe), but they couldn’t change it. They couldn’t monitor it, support it, or recognize problems when they occur sneakily. Sean is working in a dangerous silo. And he just bought a fast car.

Sean's new Honda S2000

He doesn’t want to be in a silo; a team with one or two more people would be ideal. This kind of ninja doesn’t have to work alone. There is a balance between programs and software. For many business departments, that balance may be far closer to the “program” side than software developers like me care to admit.

May I please have inlining for small dependencies?

Franklin Webber[1]  went on a quest to find out whether
utility code was better or worse than core code in open source
projects. Utility code is defined as anything necessary but not directly related to the application’s
mission. Franklin found that overall quality of utility code was higher, perhaps because this code changes less. He also found a lot of repetition among projects, many
people doing the same thing. A function to find the home directory
in several possible env vars, for instance. HashWithIndifferentAccess,
which doesn’t care whether you pass the key as string or symbol (keyword
in Clojure). He asks, why not break these out into tiny little gems?

Dependency management is an unsolved problem in our industry. We want
single-responsibility, and we want re-use. Reuse means separately
released units. But then when we compile those into one unit, transitive
dependencies conflict. Only one version of a particular library can make it into the final executable. In the example below, the Pasta library might not be compatible with the v2.1.0 of Stir, but that’s what it’ll be calling after it’s compiled into Dinner.

Programs like WebLogic went to crazy ends to provide different
classloaders, so that they could run apps in the same process without
this dependency conflict. That’s a mess!

We could say all libraries should all be backwards compatible. Compatibility is great thing, but
it’s also limiting. It prevents API improvements, and makes code ugly.
There’s more than API concerns as well: if the home-directory-finding
function adds a new place to look, its behavior can change, surprising
existing systems. We want stability and progress, in different places.

With these itty-bitty libraries that Franklin proposes we break out,
compatibility is a problem. So instead of dependency hell, we have chosen
duplication, copying them into various projects.

What if, for little ones, we could inline the dependency? Like
#include in C: copy in the version we want, and link whatever calls
it to that specific implementation. Then other code could depend on
other versions, inlining those separately. These transitive dependencies
would then not be available at the top level.

Seems like this could help with dependency hell. We would then be free
to incorporate tiny utility libraries, individually, without concern
about conflicting versions later.
I wonder if it’s something that could be built into bundler, or leinengen?

This isn’t the right solution for all our dependency problems: not large libraries, and not external-connection clients. Yet if we had this choice, if a library could flag itself as suitable for inlining, then microlibraries become practical, and that could be good practice.

[1] Talk from Steel City Ruby 2014
[2] Angry face:

Icon made by Freepik from

Quick reference: monads and test.check generators

Combine monads with test.check generators to build them up out of smaller generators with dependencies:

(require ‘[clojure.test.check.generators :as gen])
(require ‘[clojure.algo.monads :as m])
(m/defmonad gen-m 
  [m-bind gen/bind 
   m-result gen/return])

(def vector-and-elem
  (m/domonad gen-m
    [n (gen/choose 1 10)
     v (gen/vector gen/int n)
     e (gen/element v)]
    [v, e]))

(gen/sample vector-and-elem)
;; ([[0 0] 0] 
    [[0 -1 1 0 -1 0 -1 1] 0] 
    [[1 1 3 3 3 -1 0 -2 2] 3]
    [[8 4] 8]…

The generator here chooses a vector length, uses that to generate a vector, uses that to pick an element inside the vector, and then returns a tuple of the vector and the element. The syntax is cleaner than a lot of gen/bind and gen/fmap calls. It looks a lot like ScalaCheck.

I suspect we could define m-zero and m-plus in the monad to get :when conditions as well.

I’m working on a longer post that explains what’s going on here and why we would do it.

Stacking responsibilities

TL;DR – Support decisions with automation and information; give people breadth of responsibility; let them learn from the results of their choices.
When I started writing software in 1999, The software development cycle was divided into stages, ruled over by project management.

Business people decided what to build to support the customers. Developers coded it. Testers tested it. System Administrators deployed and monitored it. Eventually the customer got it, and then, did anyone check whether the features did any good?

These days Agile has shortened the cycle, and put business, development, and QA in the same room. Meanwhile, with all the tools and libraries and higher-level languages, feature development is a lot quicker, so development broadens into automating the verification step and the deployment. Shorter cycles mean we ask the customer for feedback regularly.

Now developers are implementing, verifying, deploying, monitoring. The number of tools and environments we use for all these tasks becomes staggering. Prioritization – when the only externally-visible deliverable is features, who will improve tests, deployment, and monitoring? We automate the customer’s work; when do we automate ours?

The next trend in development process helps with these: it divides responsibilities without splitting goals. Business works with customers, developers automate for business, and a slice of developers automate our work. Netflix calls this team Engineering Tools; at Outpace we call it Platform. Instead of handoffs, we have frequent delivery of complete products from each team.

Meanwhile, when developers own the features past production, another task emerges: evaluation of results. Automate that too! What is success for a feature? It isn’t deployment: it’s whether our customers find value in it. Gleaning that means building affordances into the feature implementation, making information easy to see, and then checking it out. We’re responsible for a feature until its retirement. Combine authority with information, and people rise to the occasion.[1]

Learning happens when one person sees the full cycle and its effects, and that person influences the next cycle. Experiments happen, our capabilities grow.

In this division of responsibilities, no one delegates decisions. Everyone shares a goal, and supports the rest of the organization in reaching that goal. The platform team doesn’t do deployments. It creates tools that abstract away the dirty details, supplying all information needed for developers to make decisions and evaluate the results. At Outpace, the Platform team is composed of people from the other development teams, so they share background and know each others’ pain. The difference is: the platform team has a mandate to go meta, to improve developer productivity. Someone is automating the automation, and every developer doesn’t have to be an expert in every layer.

The old way was like a framework: project managers take the requirements from the business, then the code from the developers, and pass them into the next step in the process. The new way is like libraries: the platform team provides what the developers need, who provide what the business needs, who provide what the customer needs. Details are abstracted away, and decisions are not.

When a developer’s responsibilities end with code that conforms to a document, it’s super hard to get incentives aligned to the larger needs. Once everyone is responsible for the whole cycle, we don’t need to align incentives. Goals align, and that’s more powerful. Remove misaligned incentives, give us a shared goal to work for, and people achieve. Give us some slack to experiment and improve, and we’ll also innovate.

[1] via David Marquet