This is not OK

After the first session at StrangeLoop, I ran downstairs, heading for the side of the lobby with the women’s restroom. I had to pee. Halfway there I looked up; it said “Men’s Restroom.” Turned around: “Men’s Restroom.” Wait a minute.

The venue converted the women’s room into a men’s to alleviate overcrowding. They did not ask the conference organizers. The ushers remarked later that it was the first time they’d ever changed the main women’s room into a men’s, while the other way around was common enough.

Trapped between two men’s rooms, I flipped out. I had to pee and this was NOT OK. An usher pointed me around the corner, where the Family Restroom was relabeled “Ladies.” It was a one-seater.

There was no line.

That day in the Family Restroom I threw a fit. Hurled my water bottle at the wall and screamed, “This is not OK!”

StrangeLoop is the most awesome of programming conferences, with less “you can use this in your day job tomorrow” and more “you can think about this for months and then it will embiggen everything you do.” Seven of the speakers at the conference were women, better than the local conferences I’ve attended this year. Thirty or forty women attended, of over a thousand. 3%.

I exited that bathroom and looked around at the sea of men, and it looked different than ten minutes before. Ten minutes ago it was expected. Now it was NOT OK.

There are women in programming. Usually there are a few other women wherever I’ve worked, something like 15%. (More at Amdocs, which was an Israeli company.) Look at the women who attend conferences, and there are fewer, around 10%. Take a highly technical, serious-about-this conference like StrangeLoop, and we’re down to 3%.  Speakers at technical conferences are somewhere between 3-10%, partly because organizers recognize that women speakers are a good influence on the industry and bring more women attendees.

The rest of the conference, I veered far from my usual mode of operation. Instead of talking among my friends and targeting mostly men to meet, I chased down women in hallways to say hello. Ines Sombra suggested over twitter a gathering for drinks that evening, and it became my mission to invite every woman I saw.

Saint Louis has a vibrant software industry, especially lately. We have a shocking variety of user groups; I’ve spoken at six this year. There’s a group for every ecosystem and favored text editor. Across the board, attendance is 30-50, yet if I see another woman I get excited. At first, people thought I was a recruiter. Extroverted female at a programming group? It was a reasonable assumption.

Monday evening at StrangeLoop, fourteen passionate women programmers sat around three pushed-together tables. We broke into conversations with the people nearest us. I had before never spoken with three other women about technical topics. It felt good! It felt really, really good. We considered starting a group for women in tech in St. Louis.

There aren’t enough women in programming. I applaud those who work to get more girls to choose STEM careers. (shout-out to Atomic Object, who sent a few of my favorite people to StrangeLoop.) There are initiatives to raise awareness of bias and sexism in the workplace. There is encouragement for women to speak at conferences. But what about the level above that?

There are speakers, and then there are respected thought leaders. The keynoters, the language designers, the authors of seminal texts. Where are the women there? Can you name one in this millenium?

Enrollment of women in computer science is decreasing. The women who do program move up to project management instead of architecture. Senior developers go home at five thirty while their husbands lead user groups. Why?

I want to find role models at the very top of our industry. I want to make talking tech with other women  commonplace. I want to encourage women to make programming a career. I want more women at StrangeLoop.

Confessions of a woman in tech

I have enjoyed being a woman in a profession dominated by men. I love being the only woman in a roomful of men. I love when they open doors for me. When they carry heavy boxes for me. When they give me a chair in a full room.

I love it even more when a man patiently answers all my new-team-member questions. When my infrastructure requests get done fastest because I call the man and ask him nicely. When I introduce myself to a random person in the coffee room and he is happy to talk to me.

I tell penis jokes and curse so that men won’t mistake me for a lady they have to tiptoe around. The stereotypical programmer – white, heterosexual, historically sexually frustrated, geeky — these men like me. That sexual tension can be an asset. Women don’t like me. I have made-up reasons for why, but I don’t know. I’m too busy hanging out with the guys.

At conferences it’s even better. I can sit down at any random table and strike up a conversation, and it’s easy because I’m a woman. Most men at conferences are pleasantly surprised and talk with me. Attention is everywhere. Speaking is even better, because then I seem extra-smart. I don’t have to prove my geek cred when my badge says “Speaker.”

I’ve been hit on at work. I don’t mind – hell, I relish in it. Call me “toots” and I’ll smile at you, as long as you are smart as hell or know your shit. I’m supposed to be offended, if not for my own sake then for other women who don’t want to be “toots.” But I like the attention.

The movement for women in technology has received lackluster support from me. As a woman, I’m supposed to be all passionate about this. Yet, deep down, I don’t want more women in my group. I don’t want more women speakers. I love being in the minority, because I love special attention.

Is this ugly? Fuck yeah it’s ugly. It is also me, it has been me for years, and I have to own it before I can move on from it. This is me owning it.

Next post, I’ll tell you the moment this changed.

RNA vs DNA: tradeoffs

RNA was the first carrier of genetic information. It is cheaper and faster to replicate, so in a volatile environment where adaptation had to be fast, this was efficient. However, RNA’s instability limits the information it can carry: longer strands lead to too-frequent errors. DNA, with its greater infrastructure, is more stable. Reproduction is more expensive, change is less frequent, and the level of complexity supported is much higher.

For prototyping, stick with tools that support rapid change. Dynamic languages and schemaless datastores are the RNA of the software prototyping world. Once your app is more stable and the level of complexity makes correct changes difficult, maybe you want a relational database. Increased structure might make change less frequent but safer.

comfort

When I snuggle with my youngest daughter, I sing her a special song called “La La Linda.” As I hold her and stroke her hair and sing, I am charging up the song by creating associations in her brain. Each snuggle links it closer to feelings of comfort and love.
When she falls on the sidewalk or is frightened by a loud noise, I sing the song again. This calls up those associations, bringing comfort.

Every time we code a pattern that is familiar to us, one that has worked for us before, we feel comfortable. Learning a new style is much harder – it feel less productive, it challenges us and makes us feel less like an expert. Yet, this is when growth occurs. Growth and comfort are at odds. [1]

Fiona Apple sings, “I’m good at being uncomfortable so I can’t stop changing all of the time.” [2] Change isn’t always growth, but lack of change is always lack of growth. So give it a try, be uncomfortable, relish in it if you can! I’ll take a feeling of learning over a feeling of expertness any day. The alternative: “you’re no good at being uncomfortable so you can’t help staying exactly the same” is stagnation.

——
[1] thanks to Robin on Facebook yesterday
[2] Fiona Apple, Extraordinary Machine

Functors: What the funk for?

For all the programmers who don’t deeply grok the lambda calculus terminology —

Say you are about to call a method on a container, and that container can give you something back of type Tweet. What you really want isn’t a Tweet, but some part of it, say Tweet.getId(). What if, instead of getting the Tweet back from the container and then calling getId() on it, you prefer to tell the container to get the id for you, and return only what you wanted? Then you need a functor.

So you make a quick function literal (tweet => tweet.getId()) that pulls the id out of a Tweet, and pass that in to the functor. The output is the same kind of container, only it holds IDs instead of Tweets. A functor isn’t a special type; it’s a function with a particular purpose.

Why would you want to do this? I used to think that too. Then…

I have an Iterable tweets, and I want to get the ID of the first element. In this example, I’m using Java with the Guava library. The function to call is Iterables.getFirst, which requires a default value in case the iterable is empty. As it happens, I have a default value for the ID, but no Tweet that contains this default ID. Short of constructing a stupid dummy Tweet that holds my default ID, I’m stuck with:

Tweet firstTweet = Iterables.getFirst(tweets, null) 
return firstTweet == null ? defaultId : firstTweet.getId() 

I have to create an identifier and then check stuff on it. This is not declarative enough or simple enough for my taste. I’d rather do this (using Java 8 lambda syntax for brevity):

return Iterables.getFirst(tweets, t => t.getId(), defaultId)

In this hypothetical example, I’m telling that getFirst function to run my function on the tweet before returning. If tweets is empty, then the function can return defaultId. So I’m supplying a default that is meaningful to me, the function is going to return the same type whether there’s a tweet or not, and everyone is happy.

But, that method doesn’t exist. And it would be a pain to an optional function argument to the interface of all the functions in Iterables. Thinking in terms of functors, I can solve this problem with functions that do exist in Iterables:

Iterables.getFirst(Iterables.transform(tweets, t => t.getId()), defaultId)

Here, transform is a functor: it applies the supplied function to the elements inside the iterable, returning an iterable of the function-output type. This means a tweet gets turned into a tweetId before getFirst sees it.

To a long-time Java dev like me, this looks inefficient. Do work on every tweet in tweets just to get the first one out differently? Ahhh, but Guava iterables are lazy! That function is not applied until somebody calls next() on the iterable returned by transform(tweetst => t.getId()). Iterables.getFirst calls next() at most once, so only the first tweet is transformed. Therefore, I have exactly what I wanted: turn that first tweet (if there is one) into an id before giving it back to me. The type of defaultId matches the element type of transform(tweetst => t.getId()).

The lesson of functors is: you don’t have to take something out of a container in order to operate on it.

In OO design, there’s a principle of “Tell, don’t ask.” Classes are supposed to tell each other what to do, rather than asking for internal data. This is an example of that — the Iterable has tweets, and I want the ID of the first one. Using the functor, I tell it “give me the result of this function applied to your tweet.” This is better encapsulation than pulling out the whole tweet and operating on it.

In this example, a functor is a function that does a transformation on data inside a context. In this example the context is an Iterable and its contents are Tweets. The transformation is a functor from Tweet -> ID. This way, an Iterable can give me back an ID, exactly what I wanted in the first place, without me ever having to see its Tweet.

Look! A functional trick can make my Java more OO than ever.


Caveat: there are many definitions for Functor out there, and different types of functors. This is one.

When is code data, and when is it code?

The first tenet of functional programming: code is data. Store it in variables, pass it as parameters, return it from methods.

This is challenging in the JVM, where all code lives inside methods on objects. Therefore, Scala passes around code in a function object. Code lives in the apply() method.

When is a function object created? It is not always clear when we’re reading Scala code: what is assigned to an ordinary method on a class? what will be executed immediately? what gets wrapped in a function object for storing and passing around?
This post describes how to get from executable code to a method; how to get from executable code or a method to a function literal; and how to get from a function literal or method back to executable code. Along the way I’ll mention three exceptions to the rule “a method is executed every time it is referenced,” show how to pass around a catch clause, and interrogate an innocent citizen.

Since all code in Scala is in a class (even scripts get wrapped), let’s start there.

Code in a class: now or later?

Expressions inside a class body are executed on instantiation as part of the default constructor. The two exceptions are methods (declared with def)and function literals. This can surprise the unwary coder.

If a block or line of code comes after val x =, then it’s getting executed once, right now. If it comes after def x =, then it gets executed later, whenever the method is called.

Say we want to create a wiretap on a citizen. The citizen is mutable. The following class will initialize the val phone only once; if Sally’s home phone number is updated, the wiretap will never see it. However, name is defined as a method; if Sally’s name changes, her wiretap’s name property will reflect this.

class Wiretap(target: Citizen) {

    val phone = { target.homePhone }

    def name = target.name
}

Scala’s Principle of Uniform Access means that users of Wiretap don’t have to care whether name and phone are methods or fields, they’re just properties.[1] But the writers of Wiretap had better be careful!

Function Literals

You can always defer execution of code by wrapping it in a function literal. There’s a straightforward way to do this, and there are sneaky ways.

Who doesn’t love rockets?

rocket operator

Create functions explicitly with a rocket. You can put these into a val if you like, or pass them to methods like List.map which expect a function type.

val isLegal = (w: Wiretap) => w.toString.length > 1

(We set the bar pretty low for a wiretap to be considered legal.)

Using the ubiquitous underscore, we can make a function literal without a rocket:

val isLegal = (_:Wiretap).toString.length > 1

What function is expected as a parameter

Inside a parameter list where a function of specific parameters is expected, we can skip the parameter-type declaration. Scala will do everything it can to turn what you put in that parameter into the expected function type.

val legalWiretaps = listOfWiretaps.filter( _.toString.length > 1 )

Scala will convert a method into a function if the method is referenced where a function of the same type is expected. This is one of three exceptions to “methods are invoked every time they are mentioned.”

object Wiretap {
   def isLegal(w : Wiretap) = w.toString.length > 1
}

listOfWiretaps.filter(Wiretap.isLegal)

Because filter expects a function of type Wiretap => Boolean, Scala recognizes that the method matches this signature and wraps the method in a function literal for you.

When you want a method treated as a function literal

You can tell Scala to treat the method as a function type — code-as-data rather than code to be executed — if you follow the method name with an underscore.[2] The underscore triggers the Scala compiler to wrap the method instead of invoking it.

scala> val targetNameFunction = sally.name _ 
                    // the near-invisible underscore is critical
targetNameFunction: () => String =

targetNameFunction is now a function that returns string. Every time it is applied, sally.name will be called again.

This is the second exception to “every time a method is referenced, it is invoked. The last one, sneakiest of all, is described in the next section.

Where the method you call sequesters your expression

In Java, when you call a method, all the parameter expressions are evaluated before they are passed in. This is eager evaluation. In Scala, the writer of a method can specify “when they pass me something, don’t bother figuring out what they passed me – just give me that code so I can evaluate it when I please.” This is called (for reasons I don’t understand) a by-name parameter. It’s cool because it lets Scala developers define our own control structures like loops and conditionals. It’s uncool because the caller of the method may not realize that she just passed in a function literal; it looks the same to her. That expression might be evaluated once, or many times, or not at all.

Here’s an evil method:

   def evil(sneakyMethodThing : => String) = {
      println(“Time to do some bad stuff”)
      sneakyMethodThing == “I didn’t do it” || sneakyMethodThing == “You can’t prove anything”
   }

When I call this with an expression that interrogates Sally…

evil( officer.interrogate(sally) )

… then in this one line, Sally might get interrogated twice!

The code officer.interrogate(sally) looks to the caller of evil like an expression that will be evaluated immediately. But really it gets wrapped up inside an object and passed in to the evil method. Yet inside the evil method, sneakyMethodThing doesn’t behave like a function object; you don’t have to use () to apply it. It behaves more like a method: every time sneakyMethodThing is referenced, it is invoked.

evil: Time to do some bad stuff.
officer: “Why did you do it?”
sally: “I don’t know what you’re talking about.”
[evil observes that this does not equal “I didn’t do it” and continues evaluating]
officer: “Why! Why!”
sally: “I told you, I don’t know!” [sobs]
[evil returns false]

Watch out for these by-name parameters, and be careful about passing in expressions (like this one) with potentially damaging side-effects.

Why should a function cover every case?

This is a strange animal. A partial function literal is one or more case statements in a block – like a chunk of a match expression. Catch expressions use these.

try {
  // some stuff
} catch {
   case ex : IllegalArgumentException => “I’m a dork”
   case ex : Throwable => “The world is over”
}

The part after the catch clause is a partial function. Check this out – you can put that piece in a value:

val standardExceptionHandling : PartialFunction[Throwable, Any] = {
   case ex : IllegalArgumentException => “I’m a dork”
   case ex : Throwable => “The world is over”
}

… and then use it:
try {
// some stuff
} catch standardExceptionHandling

That’s cool!

The other place partial functions are useful is when you want to use a pattern to extract data from the parameter to a function.

someMapOfThings.foreach {
    case (key, value) => /* do something with key and value */ }

Be careful here to use a pattern that will match every input. Otherwise, a MatchError can bite you at runtime.

The PartialFunction is a function object with a bonus method: isDefinedAt returns a boolean indicating whether the function has an answer for the supplied input. These are interesting in other ways, but this post is about syntax. It’s time to take this the other way and turn functions and methods back into executable code.

How to make it go

To wrap it up, go the other way: call methods and functions to execute the code inside of them.

Methods and by-name parameters are called every time they are referenced. Three exceptions: when they appear in a parameter list where a corresponding function object is expected; when they are followed by an underscore; and when they are received into a by-name parameter.

Functions are applied in two ways:

  1. follow the identifier with a parameter list. If the function accepts no arguments, then the parameter list is an empty set of parens. Unlike methods, you can’t leave off the parentheses.
  2. call the apply method directly

val k = (_:String).length

  1. k(“hi”)
  2. k apply “hi”

There’s a peculiar consequence to the first syntax here. Anything with an apply method is a function – including stuff like lists and strings and arrays.[3] If you follow a string with a parameter list, you’ll get a result.

scala> “Sally”(0)
res10: Char = S

Perhaps even more startling, if you pass a string where a function is expected, it’ll accept it.

scala> Range(0,5).map(“Sally”)
res11: scala.collection.immutable.IndexedSeq[Char] = Vector(S, a, l, l, y)

But that string is a partial function, so it’ll blow up if you apply it to something out of range.

Whew

Scala juggles the code around among methods, objects, and expressions to achieve its mix of imperative and functional support. Conceptually, code can be passed as data, even on the JVM. Those concepts aren’t always crystal-clear to the reader, though, so be careful.


[1] for added confusion, in a class, both methods (def) and fields (val) are implemented as methods. The val becomes a getter, essentially, that returns an invisible constant field.

[2] Scala’s error messages refer to this as “partially applied function,” but I dispute the use of this term. For instance, try to assign a method with parameters to a val:

scala> val e = Math.pow
:8: error: missing arguments for method pow in class MathCommon;
follow this method with `_’ if you want to treat it as a partially applied function

Again, saying “Math.pow _” will wrap the method in a function type and store that in the val. I think this is not partial application; @bvenners says it’s eta-expansion. @dickwall says it is an edge-case of partial application, supplying 0 arguments.

[3] Or there’s an implicit conversion that turns them into something that implements PartialFunction. Always tricky in Scala.

What is flatMap?

flatMap is a method on collections that, at a surface level, transforms each element into another collection of any type, then takes all the elements out of those collections and puts them into one. It is like a map followed by a flatten.

Why do two things in one step?

Conceptually, List.map is always 1:1 – each element in the input collection produces one element for the output collection. With flatMap, each element in the input collection produces 0 or more elements for the output collection.

This is like the map in CouchDB’s MapReduce: the map function can emit zero or more output documents for each input.

Or as @Deech says, “It’s just a monadic bind.” In slightly more colloquial terms, we can think of it as: pass a functor into a context, and get back the same kind of context with different stuff in it.

waste not, grow not

Duplication of effort. Is it a bad thing?

IT organizations put a lot of effort into reducing duplicate code and design effort. Architects are tasked with producing standards and choosing technology for the entire organization. Responsibilities are divided carefully, and watchdogs look for opportunities for reuse, increasing conformity. Data is painstakingly normalized so that the same bytes are never repeated twice.

Is this optimal? Amazon defied this, breaking teams into independent entities, with their own hiring practices. Requiring backward compatibility, letting security and authorization be implemented over and over again. How inconsistent! How inefficient! The result? AWS. A surprise, and now very big business for what used to be a retailer.

The economy is an example. It is obscene duplication of effort for the same product to be produced in two factories near each other, with minimal differences in output. The same decisions made again and again, reduced economies of scale. If what you care about is reducing duplication of effort, central control is clearly superior.

Oh wait – communism doesn’t work. Centrally directed economies fail miserably. Instead, capitalism, with all kinds of random companies making the same decisions in different ways, duplicating effort all over the place but never quite the same, rising and falling unpredictably – all this leads to phenomenally better results. It leads to invention. To surprise.

It is like: say you have some blocks. To cover the most ground, you reduce overlap. Overlap is duplication of effort. But what do you get if you allow overlap? Height!

Perhaps if all the architect blocks running around making sure the other blocks don’t overlap, flattening them out, were instead allocated along with the others to arranging themselves as they found most fulfilling, the blocks would cover just as much ground and also create a pyramid.

Control reduces potential.

mastery

They say ten thousand hours of practice is what is takes to master a skill. Ten thousand hours of playing violin, of hockey, of programming. No: not ten thousand hours of doing it. Ten thousand hours of conscious practice with the intent to get better. [1]

When riding a motorcycle, devote 5% of your attention to recording what you do and what the results are. [2] Later, reflect on “I leaned this far, and the bike turned in the center of the lane. If I lean farther, I can take it tighter.”

As professional developers, we have the chance to put in these hours and get paid for it. If we code eight hours a day, that’s only five years. That is unrealistic, so call it ten. After ten years, are we masters of our craft? Only if we code every day with the conscious intent to get better. Are you learning from your work every day? Thinking about what you’re doing and how to do it better tomorrow?

This is the difference between ten years of experience and the same year of experience ten times.

————–
[1] Outliers, by Malcolm Gladwell, ch 2
[2] this one book we have at home about how to get good at riding motorcycles, like for racing.

we do not exist in isolation

People, like code, do not operate indeoendently of their environment.

An isolated town of shockingly healthy individuals [1] showed that our health depends not only on factors intrinsic to us (genetics) or controlled by us (diet and exercise). Health is hugely influenced by community. We can’t be understood in isolation.

When a bug turns up in test or production, people love to ask “what did we change? That last code change must be the cause!” No. Lots of things changed. Different users from last week are typing in different input. The contents of the database are not identical. Perhaps the hardware has changed, or some downstream system is reacting differently. When a problem occurs, we should look at the problem, dig down until we know what happened, and then we shall find the cause. Sometimes it is the code change, and by finding it by understanding the problem, we will know how to fix it. (By something other than knee-jerk “back it out!”)

Programs and people are both defined not only by internal state, but also by the world around them.

——–

[1] Outliers, by Malcolm Glad well, preface.

[2] This makes a case for functional programming. Functions that are referentially transparent, or “data in, data out”, do not access external state. Output depends only on input. Results are the same every time, for the same input. This is testable. Write as much of your code as possible in this style, and the fragile parts will be isolated. Purely referential code is useless in the real world, but it is easier to test.