TL;DR: the most productive development happens when one person knows the system intimately because they wrote it; this is in conflict with growing a system beyond what one person maintains.
Let’s talk about why some developers, in some situations, are ten times more productive than others.
hint: it isn’t the developers, so much as the situation.
When do we get that exhilarating feeling of hyperproductivity, when new features flow out of our fingertips? It happens when we know our tools like the back of our hands, and more crucially, when we know the systems we are changing. Know them intimately, like I know the contents of my backpack, when I packed it and I tuned the items in each pouch over years of travel. Know the contents of every module, both what they are and what we’d like them to be if we ever finish that refactoring. Know the edges, who uses every API and which changes will break whom, and we’re friends with all of the stakeholders. Know the underpinnings, which database fields are indexed and which are obsolete and which have quirky special values. Know the infrastructure, where it runs in production and how to ssh in; where it runs in test and what version is deployed and when it is safe to push a new one. Know the output, what looks normal in the logs and what’s a clue. We have scripts, one-liners that tail the logs in all three prod instances to our terminals so our magic eyes can spot the anomaly.
We know it because we wrote it, typically. It is extremely difficult to establish this level of intimacy with an existing system. Braitenberg calls this the Law of Downhill Invention, Uphill Analysis. Complex systems are easier to build than to figure out after they’re working.
We know it because we are changing it. The system is alive in our head. It’s a kind of symbiosis: we help the system run and grow, and the system works the way we wish. If we walk away for a month or two, begin a relationship with a different system, the magic is lost. It takes time to re-establish familiarity.
Except, I’m suspicious of this description. “We” is a plural pronoun. This depth of familiarity and comfort with a system is personal: it’s usually one person. The one person who has conceived this solution, who holds in their head both the current state and where they are aiming. (sometimes known as the purple developer)
If you are this person, please realize that no one else experiences this work the way you do. For other people, every change is scary, because they don’t know what effect it will have. They spend hours forming theories about how a piece works, and then struggle to confirm this with experiment; they don’t have the testing setup you do. They study every log message instead of skimming over the irrelevant ones, the ones you’ve skipped over so often you don’t even see them anymore. By the time they do figure something out, you’ve changed it; they can’t gain comprehension of the system as quickly as you can alter it. When they do make a change, they spend lots of time limiting the scope of it, because they don’t know which changes will cause problems. They get it wrong, because they don’t know the users personally; communication is hard.
If you are this person, please go easy on everyone else. You are a synthetic biologist and can alter the DNA of the system from within; they are xenosurgeons and have to cut in through the skin and try not to damage unfamiliar organs.
If you work with this person, I’m sorry. This is a tough position to be in, to always feel inferior and like you’re breaking everything you touch. I’m there now, in some parts of our system. It’s okay for me because the host symbiont, the author and manipulator of that software, is super nice and helpful. He doesn’t expect me to work with it the same way he does.
If your team looks like this, here are some steps to take:
- Consider: don’t change it. This really is the fastest way to develop software. One person, coordinating with no other developers, can move faster than a whole team when the system is small enough. Until! we need the system to grow bigger. Or! the system is crucial to the business (it’s an unacceptable risk for only one person to have power over it).
- As the host symbiont who lives and breathes the system: strike the words “just”, “easy,” “obvious,” “simple,” and “straightforward” from your vocabulary. These words are contextual, and no other human shares your context.
- Please write tests. Tests give people who are afraid of unintentional breakages a way to test their theories. Experimentation is crucial to learning how a system works, and tests make experiments possible. They also serve as documentation of intended behavior.
- Pair program! By far the best way to transfer understanding of the system to another human is to change it together. (Or boost a whole team at once: mob program!)
- Make a README for other developers. Describe the purpose of the system briefly, and document how you develop, test, and troubleshoot the system. Specify the command lines for running tests, for deployment, for accessing logs. Describe in detail how to obtain the necessary passwords. Write down all the environments where it runs, and the protocol around changing them.
- Do you know your users better than anyone else? Remedy that. Bring other team members into the discussion. (There’s a sweet spot of a single developer-type who works within a business unit. When the software becomes too important for this scale, it gets harder.) Let all the devs get to know the users. Have happy hours. Form redundant communication channels. It’ll pay off in ways you never detect.
- Slow down. Like seriously, if one person is developing at maximum speed on a project, no one else can get traction. You can’t move at full speed and also add symbionts. When it is important to bring in new people, don’t do anything alone. Pair on everything. Yes, this will slow you down. It will speed them up. Net, we’ll still be slower than you working alone. This is an inherent property of the larger system, which now includes interhuman coordination. There’s more overhead than when it was just you and your program. That’s OK; it’s a tradeoff for safety and scale from sheer speed.
Let’s acknowledge that there really are developer+situations that are 10x more productive than others. Let’s acknowledge that they don’t scale. Make choices about when we can take advantage of the sweet spot of local or individual automation, and when the software we’re building is too important for a bus factor of one.
Distinguish between an experimental prototype, when speed of change and redirection is crucial, versus a production app which needs backwards compatibility guarantees and documentation and all that seriousness — this requires a solid team.
Recognize that the most productive circumstance for development is a rare circumstance. Most of the time, I need to work on a system that someone else wrote. (that “someone else” could be “me, months or years ago.”) The temptation to rewrite is strong, because if I rewrite it then I’ll understand it.
There’s a time for 10x development, and a time for team development. When you want to be serious, the 10x developer prevents this. If that’s your situation, please consider the suggestions in this post.
Excellent observation, Jessica. I'm curious whether you — with your strong experience in OO and functional — would agree that what you wrote is especially true of OO systems and less true of functional systems. To me, functional systems are inherently simpler to understand, debug, etc. because code paths are easier to trace and places where state can change are easier to locate. The modularity of functional code means you don't have to cram a model of the entire codebase into your head to debug or extend that code, as you must with big-ball-of-mud OO monoliths. Functional code's no-magical-side-effects constraint makes it easier to grok and is one reason I prefer functional.
I have fewer questions about what will happen when I call a given (pure) function with a given set of values, but comprehending a system is still non-trivial.
immutability takes some speedbumps and surprises out, true.There are code styles and faux pas (in both OO and FP) that make code harder to learn, but like Donald said, nothing can make it easy.
I wanted to disagree but then I realized I've been teaching teams to reign in and break these wild stallions with heavy test coverage and documentation requirements for years. It didn't really occur to me that we could just ask the high performer to take steps to not leave the others in the dark.
You read my mind 🙂 and yes, I have been that person (both of them) many times, so I get it, and I can't stress enough the value of having good documentation and solid coding practices. Thanks!
It helps if the code was originally written in a way that the structure is easy to figure out, since \”structure is the key to understanding.\” Nice post.
I love this article and it's very relevant to recent experience. (Well, I suppose it's probably relevant to most programmers most of the time!) I've found it difficult to convey these points to upper management and other stakeholders less involved in the day-to-day of software development. An antipattern can develop in startups where the founding leadership can't understand why things don't move as quickly as they did when the company was three people in a room and the age of the codebase was measured in weeks or months, and ends up concluding that it's because the current devs are a bunch of slackers and/or incompetents compared to the original architect who knew every line like the back of their hand.
Fully recognize your points. Often, teams go slow due to significant amounts of legacy. The more code you own and need to maintain, the slower you go. If you continue to add features, your productivity moves to zero. Our solution direction involves the use of model driven engineering, including model mining from legacy code.I explain my thinking the article below:https://www.linkedin.com/pulse/gardeners-dilemma-dirk-jan-swagerman
This is a great article. Have you ever had to deal with a team after the inventor has left? I am working through that right now, and would love some insight.
I agree that large systems and maintaining systems are harder than small systems and growing systems. I also agree, strongly, that the tests are what makes or breaks the ability for a team to make forward progress over time.However, there are still differences between engineers in their ability to comprehend and make effective changes to large systems they didn't build and haven't memorized. The difference between a low-performing developer and a high-performing developer, when starting from the same scratch position, can still be 10x. If you then compare junior-developer-in-new-big-system to high-productivity-developer-in-small-known-system, you may see, at the limit, a 100x difference in effectiveness.
So, i been thinking about this concept of net present value discounted lifetime cost of a line of code. Jessica correctly points out that depending on 10x developers does not scale. And while a developer maybe 10x initially, is he or she going to be 10x on the NPV discounted cost of ownership of that line of code?Good teams will have a code down target and make sure they reduce their inventory:https://www.linkedin.com/pulse/why-your-team-needs-code-down-target-dirk-jan-swagerman
Great observations here! Enjoying discussing this article with my team.In addition to the techniques Jessica outlines, we derive a lot of benefit from code reviews as a medium for teaching/onboarding onto systems. I'd say we employ code review similarly to how pair programming is advised in steps 4 and 7. (We also pair, sometimes!)Another overlapping topic that might be fruitful to compare is a spectrum of schemes of code ownership https://martinfowler.com/bliki/CodeOwnership.html and how they might make the process of onboarding/contributing to a new subsystem easier or harder.
I've been working on this problem for a while: conveying the global structure of a project to others, and helping outsiders to ramp up and gradually become insiders. Details: http://akkartik.name/about
Hi my name is Brittany and I'm the Marketing Associate at PowerToFly. We are a company that works to elevate and empower women in tech with jobs and the resources they need to succeed. We love this piece and were wondering if you like to repost it on our blog targeting women in tech, with proper credit of course. Looking forward to your response!
Hi my name is Brittany and I'm the Marketing Associate at PowerToFly. We are a company that works to elevate and empower women in tech with jobs and the resources they need to succeed. We love this piece and were wondering if you like to repost it on our blog targeting women in tech, with proper credit of course. Looking forward to your response!
Quoting http://antirez.com/news/112#comment-3180320056:There's another, more common way to be a 10x programmer: generate technical debt at a rate that requires ten other developers to clean up the mess. The real magic is that this feeds back on itself: you look more productive while producing garbage, and everyone else is less productive, because they're bogged down in your disaster. Management then directs more resources to you, and less to everyone else, since you're a '10x programmer' who is 'getting things done'.Fuck those guys.
\”Know them intimately, like I know the contents of my backpack, when I packed it and I tuned the items in each pouch over years of travel.\” The follow-up, care to share the layout of your backpack? Do you prefer more pockets and pouches? Do you customize, (Make additional pouches with a sowing machine)?In regard to the rest … The mythical man month, brooks law. Your point \”number seven\” to slow down is interesting advice. When developing open source, I found benefit in slowing down. We had a core team, (more than one architect). If a core team member went too far developing one feature he became the \”sole architect\” for that feature. This slowed down development for others developing in that area. We found slowing down and pair programming to finish up commits ensured we were on the same page. You would also see this in a code review. Where design was questioned and changed. In that case the whole team agrees on the design direction. Result was a definite benefit across the team to efficiency when building the project. Because we all became opinionated and of the same opinion on the direction of the code.
Really Good blog post.provided a helpful information about most productivec ircumstances for html .keep updating… Digital marketing company in Chennai
I agree that productive developers should take time to help keep the team productive and not just disappear into their own little silo. A little mentoring can go a long way. But it's inaccurate to say that they are more productive just because they are familiar with the application. In many cases it's experience. Perhaps the 10x developer writes unit tests while the rest resist and then spend more time debugging and putting out fires. I'm not saying there's one answer that applies in every situation. But let's not go all Harrison Bergeron and blame experienced, productive developers when others are less productive.
I don't think Jessica is talking about blame here, or dragging a productive developer down to less productive developers' levels. It's more about using that 10x developer's mojo to lift the whole team up, and improve the team's velocity in the longer term.
I was reading this thinking, \”This whole experience is really foreign, and I'm pretty sure it's because we pair all the time.\” I was pleased to see that (and automated testing) in the practical application. I guess maybe we have some 2x or 3x developers who are constantly kept in check by their team, but also constantly pulling their team forwards. When's everything is paired on, most knowledge is transferred, every design is verbalised, and every line of code is understandable to at least two people when written. Unfortunately the large costs of pairing are immediate while the biggest benefits are medium-long term, so many orgs never end up finding out what they're missing.
I'd read and heard this concept of the 10X developer, and it just clicked for me in a completely different way after reading your posts. It also explains for me something from my own past experience that now for the first time totally makes sense. I was working as an engineer writing the embedded software for a digital oscilloscope. Because of the nature of the instruments we were making, not only was all the software custom-built, but so was most of the hardware. I got started pair programming with another developer who was not only extremely knowledgable, but also very patient. One of the custom chips we had to interface with was actually designed by the guy I carpooled into work with every day. Any question I had about it, I could ask him, and he would have answers because he knew his chip intimately. I would fix my code, and sometimes even, he would go back and make changes to the chip in response to issues that I would bring up on the software side. The manufacturing floor where the instruments were assembled and tested was right downstairs, and if something wasn't working right, they'd come up to let us know right away. Many of the hardware and software engineers on this team had been there for many years, and knew the system through multiple generations of products. This setup was extremely productive. Every place I've worked at since was a bigger operation, with higher employee turnover. It's clear that no number of developers or sophisticated development tools can substitute for a team that knows its product intimately.
A really good article, although I do disagree a bit with:\”We know it because we wrote it, typically. It is extremely difficult to establish this level of intimacy with an existing system. Braitenberg calls this the Law of Downhill Invention, Uphill Analysis. Complex systems are easier to build than to figure out after they're working.\”In my experience, it is more about attitude, and it is not really more difficult to understand a system built by someone else. Put it this way: someone smart enough and experienced enough to write a piece of quality software in the first place (and, really, not many people actually know how to do that) will have no trouble sitting down and systematically understanding a piece of software someone else wrote.I think otherwise, you make a lot of interesting points.