Nested learning loops at Netflix

Today in a keynote at Spring One, Tom Gianos from Netflix talked about their internal data platform. He listed several components, ending with quick mention of the “Insights Services” team, which studies how the platform is used inside Netflix. A team of people that learns about how internal teams use an internal platform to learn about whatever they’re doing. This is some higher-order learning going on.

It’s like, a bunch of teams are making shows for customers. They want to get better at that, so they need data about how the shows are being watched.

So, Netflix builds a data platform, and some teams work on that. The data platform helps the shows teams (and whatever other teams, I’m making this up) complete a feedback loop, so they can get better at making shows.

diagram: customers get shows from the show team; that interaction sends something to the data platform, which sends something to the shows team. That interaction (between the shows team and the data platform) sends something to the Insights Services team, which sends info to the data platform team.

Then the data platform teams want to make a better data platform, so an Insights Services team collects data about how the data platform itself is used. I’m betting they use the data platform for that. I also bet they talk to people on the shows teams. Then Insights Services closes that feedback loop with the data platform team, so that Netflix can get better at getting better at making shows.

Essential links in this loops include telemetry in all these platforms. The software that delivers shows to customers is emitting events. The data platform jobs are emitting events about what they’re doing and for whom.

When a human does a job, reporting what they’re doing is extra work for them. (Usually flight attendants write drink orders on paper, or keep them in memory. The other day I saw them entering orders into iPads. Guess which was faster.) In any human system, gathering information costs money, time, and customer service. In a software system, it’s a little extra network traffic. Woo.

Software systems give us the ability to study them. To really find out what’s going on, what was working, and what wasn’t. The Insights Services team, as part of the data platform organization, can form hypotheses and then test them, adding telemetry as needed. As a team with internal customers, they can talk to the humans to find out what they’re missing. They can get both the data they think they need, and a glimpse into everything else.

Software organizations are a beautiful opportunity for learning about systems. We can do science here: a kind of science where we don’t try to find universal laws, and instead try to find the forces at work in our local situation, learn them and then sometimes change them.

When we get better at getting better — wow. That adds up to some serious acceleration over time. With learning loops about learning loops, Netflix has impressive and growing advantages over competitors.

Moving beyond simplicity

We prefer simple models to complicated ones. Circles to ellipses, a single ancestor to a soup. But is that really because the simple explanation is more likely?

The value of keeping assumptions to a minimum is cognitive.

Philip Ball, The Tyranny of Simple Explanations

Simpler theories are more useful because we can think about them more easily. We can pass them from person to person.

But now we have software, which can encapsulate complicated theories almost as easily as simple ones. We can package them up into products and make use of them, without taking up brainspace with all those complications.

Maybe you still find simple models most likely in physics. Simple models are not accurate in biology – much less in human interactions.

In biology, everything going on had some reason it got that way (might be randomness) and many reasons that it stays that way. Most adaptations are exaptive — used for some purpose other than their original purpose. Most have many purposes, the proteins participate in multiple pathways, the fur is warm and also distinguishing, the brain anticipates danger and builds social structure.

In our own lives, every action has multiple stories. I’m playing this game to relax, I’m playing this game to avoid writing, I’m playing this game to learn from it.

We choose based on probability spaces from many models, stacked on top of each other. I buy this dress because I feel powerful in it, because I deserve it, because my date will like it, because I am sloppy with money. I value this relationship because I don’t know how to be alone, because they’re a kind person, because we belong together, because he reminds me of the father figure from my childhood.

This is right, this is healthy, to have all these stories. In real life, the simple explanation is never complete. Taking action for exactly one reason is unnatural. Like gaming a metric, it fails to account for the whole rest of the system.

Everything we do in a complex system has many effects, so it is right that we have many overlapping reasons. This doesn’t make life easy, only real.

When I started in software I loved that it was simple and predictable. Now I love that it is complex and messy, because that’s where I learn the important stuff.

Soft, or hard like mud

Soft skills are hard. “They take work to build and work to apply.”

@ruthmalan

The word “hard” describes sciences like physics and chemistry. It is confusing that “hard” can mean difficult, because these sciences aren’t more difficult than the “soft” ones like sociology and anthropology. They’re differently difficult.

The “hard” sciences are hard because they’re solid. We can stand on them. Physical laws are universal laws. They demand rigor, and rigor means proving that assertions apply in all cases, both through deduction and by checking against evidence.

The “soft” sciences are differently hard. They study humans systems, far too complex to make universal causal predictions. Conclusions about one culture or group are valid in some other groups and invalid in others. Rigor in complexity means studying which cases your assertions apply to. It means observing, discerning, and wallowing in context.

I describe my career progression from solving puzzles to growing products like this:

Correctness, puzzles, and the “hard” sciences are hard like rocks. Rock climbing is very technical. It takes a lot of skill and strength and hard work to climb. At the top of the cliff, you know you’ve achieved something specific.

Change, people, and the “soft” sciences are hard like mud. Like wading through goopy mud. Techniques can help, but each depends on the kind of mud. Strength helps, but pushing too hard can get you more stuck. It always helps to be okay with getting messy. When you finally reach that piece of relatively solid ground, the view is still a swamp.

Why would you want to wade through mud?

why does a fish want to swim through water?

People, interrelationships, change — this is our world. Sometimes we can carve out puzzles we can solve for real. The few universal laws we can find are priceless. But the rest of it — life is in the mud, in the deep context. The skills to navigate it are not easy. They are not satisfying in the same way, either. But they are how we find meaning, how we participate in a system bigger than our own self.

I am okay with getting messy.

Develop before define

First the loose thinking and the building up of a structure on unsound foundations and then the correction to stricter thinking and the substitutions a new underpinning beneath the already constructed mass.

Gregory Bateson on the advance of science. (From Steps to an Ecology of Mind)

This expresses a process I have observed in developers. We can develop something faster than we can define it.

That loose thinking includes the construction of loose code. We think with our fingers and eyes, keyboards and screens, editors and runtimes as well as with our brains. We try things, we draw them out or code them up. This eliminates a lot of impossible paths.

Then afterward, we shore up the useful ones. We put an API around it, error handling within, types throughout. We describe its interface and action in documentation.

Bateson grants permission to code loosely as an extension to thinking loosely, with the responsibility to return with rigor before we rope in other teams.

So do this, play in code the way we play in thought.

Then please realize that putting the foundations under it, defining the functionality so others can use it, is 10-100 times more time-consuming than your happy-path sketch.

Domain-specific laws

“there appear new laws and even new kinds of laws, which apply in the domain in question.”

David Bohm, quoted by Alicia Juarrero

He’s talking about the qualitative transformation that happens in a system when certain quantitative transition points are passed.

Qualitative transformation

I notice this when something that used to be a pain gets easier, sufficiently easier that I stop thinking about it and just use it. Like git log. There is such a thing as svn log but it’s so slow that I used it once ever in my years of svn. The crucial value in git log is that it’s so fast I can use it over and over again, each time tweaking the output.

  • git log
  • git log --oneline
  • git log --oneline | grep test
  • etc.

Now git log has way more functionality, because I can combine it with other shell commands, because it’s fast enough. This changes the system in more ways than “I use the commit log”: because I use the log, I make more commits with better messages. Now my system history is more informative than it used to be, all since the log command is faster.

The REPL has that effect in many languages. We try stuff all the time instead of thinking about it or looking it up, and as a result we learn faster, which changes the system.

Non-universal laws

I love the part about “laws, which apply in the domain in question.” There are laws of causality which are not universal, which apply only in specific contexts. The entire system history (including all its qualitative transformations) contribute to these contexts, so it’s very hard to generalize these laws even with conditions around them.

But can we study them? Can we observe the context-specific laws that apply on our own team, in our own symmathesy?

Can we each become scientists in the particular world we work in?

Which came first, the chicken or the egg?

a telling question.

This puzzler says something about our culture. It says we think in terms of causes that happen before their effects. That we don’t believe in reflexive causality.

In life, everything interesting is a circle. The mitochondria breaks down sugar, the proteins use the energy, they keep up the cell wall, the cell wall protect the mitochondria. The thriving of each one feeds the thriving of the others. I feel cranky, so I work with less patience, so I get way more errors, so I get more cranky. There aren’t many women in programming, women don’t picture themselves in programming, there are fewer women in programming. Poverty eats time and brainpower, leaving less for study and thinking, and stereotypes reinforce themselves. Suspect an employee is underperforming, and you can drive them to underperform. Or a nation’s economy is tight, citizens get paranoid and vote to increase barriers, their economy shrinks. A family helps each other, becomes a tighter and more successful unit, and helps each other even more.

Circles like these form systems. Enduring systems like a family, or short-lived whirlpools like me being cranky. Biologically, these emerge as organisms and ecosystems. As people, organizations and religions and markets and successful companies emerge from circles of mutual interdependence.

most causality is circular. Example: when I go to conferences, it means I learn cool stuff by talking to people, which means I think of interesting things to research and talk about (extra help from conversations with people I met), which means I get invited to speak (extra help for learning cool stuff at the speakers dinner), which means I go to conferences; circle repeats. Emergent property: Jessitron, international speaker.

Love has a circular causality. I find you interesting, that feels good, which makes you like me, which makes me more interesting, which feels good to me, so I like who I am with you, so I tell you sweet things. This cycles into a relationship.

There is no single cause of this phenomenon. That spark of compatibility when we met, we can point to a beginning in this case, to a “before.” but our whole life together wasn’t there. We built that moment by moment in the feedback loop of caring for each other.

“Which came first, the chicken or the egg?” is a fallacy: the fallacy of root cause. Of linear causality.

Linear causality happens, but in real life, it is the exception. An edge case. Everything interesting is a circle.

Special Relativity finally makes sense

Preface: I studied physics in college, yet there were parts of it that didn’t click with me. Special relativity with all its “a flashlight on a train” examples, that’s counterintuitive and hard. Yesterday in Surfaces & Essences, the authors explain Einstein’s thought process when he came up with it, and it makes sense this way. It’s also an interesting example about how genius can happen when we let go of what we know.

You know how when you’re on a plane, travelling smoothly at 600 mph, you can pour water the same way as when you’re standing still? If you throw a ball straight up, it falls back down into your hand.

This is called “Galilean Relativity.” It says that while you’re traveling at a constant velocity, mechanical experiments behave the same as at any other velocity (including standing still). It’s only changes in velocity — speeding up, slowing down, bumps and curves — that affect the motion of objects you have with you. This means there is no mechanical experiment you can perform to find out whether you’re flying at 600 mph or sitting on the tarmac. The water and the ball behave the same.

Einstein said, why should electromagnetic experiments be any different? Why should a lens, a magnet, or an electron care about your velocity? Your computer works the same in the air as on the ground, and there are a lot of electrons moving around in there.

The behavior of light passing through a lens or electricity moving through your processor is closely related to the constant speed of light. For your glasses and your computer to behave exactly the same in a moving plane, the speed of light inside the plane must be the same as the speed of light on the ground. The electrons in your computer move the same way, relative to you, at 600 mph or at a stop.

Speed is distance divided by time. The distance that water traveled between the pitcher and the cup looks different to you and to the watchtower, since the watchtower sees the plane moving horizontally. Same with the light from your monitor: it travels a different distance from the tower’s perspective than yours. If the distances are different, how can the speed be the same? Speed can be the same if time is different too.

Therefore time is stretched or squashed depending on velocity, so that the speed of light is the same from every perspective. No experiment can tell the difference, and electrons in your computer don’t care how fast you’re going.

Next time you sip a drink on your way across the Atlantic, appreciate time dilation. You may not notice the fraction of a second this trip adds to your life, but reliable electronics are important at every speed.

In this story, Einstein takes one belief about the world (Galilean Relativity) and broadens it. In doing so, he narrows  another belief about the world (consistent time). The result is a new, self-consistent worldview. This worldview has proven useful.

It pays to be specific about how we know something is true, and the range of experience we can be sure about.