Sunday, September 30, 2012

This is not OK

After the first session at StrangeLoop, I ran downstairs, heading for the side of the lobby with the women's restroom. I had to pee. Halfway there I looked up; it said "Men's Restroom." Turned around: "Men's Restroom." Wait a minute.

The venue converted the women's room into a men's to alleviate overcrowding. They did not ask the conference organizers. The ushers remarked later that it was the first time they'd ever changed the main women's room into a men's, while the other way around was common enough.

Trapped between two men's rooms, I flipped out. I had to pee and this was NOT OK. An usher pointed me around the corner, where the Family Restroom was relabeled "Ladies." It was a one-seater.

There was no line.

That day in the Family Restroom I threw a fit. Hurled my water bottle at the wall and screamed, "This is not OK!"

StrangeLoop is the most awesome of programming conferences, with less "you can use this in your day job tomorrow" and more "you can think about this for months and then it will embiggen everything you do." Seven of the speakers at the conference were women, better than the local conferences I've attended this year. Thirty or forty women attended, of over a thousand. 3%.

I exited that bathroom and looked around at the sea of men, and it looked different than ten minutes before. Ten minutes ago it was expected. Now it was NOT OK.

There are women in programming. Usually there are a few other women wherever I've worked, something like 15%. (More at Amdocs, which was an Israeli company.) Look at the women who attend conferences, and there are fewer, around 10%. Take a highly technical, serious-about-this conference like StrangeLoop, and we're down to 3%.  Speakers at technical conferences are somewhere between 3-10%, partly because organizers recognize that women speakers are a good influence on the industry and bring more women attendees.

The rest of the conference, I veered far from my usual mode of operation. Instead of talking among my friends and targeting mostly men to meet, I chased down women in hallways to say hello. Ines Sombra suggested over twitter a gathering for drinks that evening, and it became my mission to invite every woman I saw.

Saint Louis has a vibrant software industry, especially lately. We have a shocking variety of user groups; I've spoken at six this year. There's a group for every ecosystem and favored text editor. Across the board, attendance is 30-50, yet if I see another woman I get excited. At first, people thought I was a recruiter. Extroverted female at a programming group? It was a reasonable assumption.

Monday evening at StrangeLoop, fourteen passionate women programmers sat around three pushed-together tables. We broke into conversations with the people nearest us. I had before never spoken with three other women about technical topics. It felt good! It felt really, really good. We considered starting a group for women in tech in St. Louis.

There aren't enough women in programming. I applaud those who work to get more girls to choose STEM careers. (shout-out to Atomic Object, who sent a few of my favorite people to StrangeLoop.) There are initiatives to raise awareness of bias and sexism in the workplace. There is encouragement for women to speak at conferences. But what about the level above that?

There are speakers, and then there are respected thought leaders. The keynoters, the language designers, the authors of seminal texts. Where are the women there? Can you name one in this millenium?

Enrollment of women in computer science is decreasing. The women who do program move up to project management instead of architecture. Senior developers go home at five thirty while their husbands lead user groups. Why?

I want to find role models at the very top of our industry. I want to make talking tech with other women  commonplace. I want to encourage women to make programming a career. I want more women at StrangeLoop.

Confessions of a woman in tech

I have enjoyed being a woman in a profession dominated by men. I love being the only woman in a roomful of men. I love when they open doors for me. When they carry heavy boxes for me. When they give me a chair in a full room.

I love it even more when a man patiently answers all my new-team-member questions. When my infrastructure requests get done fastest because I call the man and ask him nicely. When I introduce myself to a random person in the coffee room and he is happy to talk to me.

I tell penis jokes and curse so that men won't mistake me for a lady they have to tiptoe around. The stereotypical programmer - white, heterosexual, historically sexually frustrated, geeky -- these men like me. That sexual tension can be an asset. Women don't like me. I have made-up reasons for why, but I don't know. I'm too busy hanging out with the guys.

At conferences it's even better. I can sit down at any random table and strike up a conversation, and it's easy because I'm a woman. Most men at conferences are pleasantly surprised and talk with me. Attention is everywhere. Speaking is even better, because then I seem extra-smart. I don't have to prove my geek cred when my badge says "Speaker."

I've been hit on at work. I don't mind - hell, I relish in it. Call me "toots" and I'll smile at you, as long as you are smart as hell or know your shit. I'm supposed to be offended, if not for my own sake then for other women who don't want to be "toots." But I like the attention.

The movement for women in technology has received lackluster support from me. As a woman, I'm supposed to be all passionate about this. Yet, deep down, I don't want more women in my group. I don't want more women speakers. I love being in the minority, because I love special attention.

Is this ugly? Fuck yeah it's ugly. It is also me, it has been me for years, and I have to own it before I can move on from it. This is me owning it.

Next post, I'll tell you the moment this changed.

Saturday, September 22, 2012

Functors: What the funk for?

For all the programmers who don't deeply grok the lambda calculus terminology --

Say you are about to call a method on a container, and that container can give you something back of type Tweet. What you really want isn't a Tweet, but some part of it, say Tweet.getId(). What if, instead of getting the Tweet back from the container and then calling getId() on it, you prefer to tell the container to get the id for you, and return only what you wanted? Then you need a functor.

So you make a quick function literal (tweet => tweet.getId()) that pulls the id out of a Tweet, and pass that in to the functor. The output is the same kind of container, only it holds IDs instead of Tweets. A functor isn't a special type; it's a function with a particular purpose.

Why would you want to do this? I used to think that too. Then...

I have an Iterable<Tweet> tweets, and I want to get the ID of the first element. In this example, I'm using Java with the Guava library. The function to call is Iterables.getFirst, which requires a default value in case the iterable is empty. As it happens, I have a default value for the ID, but no Tweet that contains this default ID. Short of constructing a stupid dummy Tweet that holds my default ID, I'm stuck with:

Tweet firstTweet = Iterables.getFirst(tweets, null) 
return firstTweet == null ? defaultId : firstTweet.getId() 

I have to create an identifier and then check stuff on it. This is not declarative enough or simple enough for my taste. I'd rather do this (using Java 8 lambda syntax for brevity):

return Iterables.getFirst(tweets, t => t.getId(), defaultId)

In this hypothetical example, I'm telling that getFirst function to run my function on the tweet before returning. If tweets is empty, then the function can return defaultId. So I'm supplying a default that is meaningful to me, the function is going to return the same type whether there's a tweet or not, and everyone is happy.

But, that method doesn't exist. And it would be a pain to an optional function argument to the interface of all the functions in Iterables. Thinking in terms of functors, I can solve this problem with functions that do exist in Iterables:

Iterables.getFirst(Iterables.transform(tweets, t => t.getId()), defaultId)

Here, transform is a functor: it applies the supplied function to the elements inside the iterable, returning an iterable of the function-output type. This means a tweet gets turned into a tweetId before getFirst sees it.

To a long-time Java dev like me, this looks inefficient. Do work on every tweet in tweets just to get the first one out differently? Ahhh, but Guava iterables are lazy! That function is not applied until somebody calls next() on the iterable returned by transform(tweetst => t.getId()). Iterables.getFirst calls next() at most once, so only the first tweet is transformed. Therefore, I have exactly what I wanted: turn that first tweet (if there is one) into an id before giving it back to me. The type of defaultId matches the element type of transform(tweetst => t.getId()).

The lesson of functors is: you don't have to take something out of a container in order to operate on it.

In OO design, there's a principle of "Tell, don't ask." Classes are supposed to tell each other what to do, rather than asking for internal data. This is an example of that -- the Iterable<Tweet> has tweets, and I want the ID of the first one. Using the functor, I tell it "give me the result of this function applied to your tweet." This is better encapsulation than pulling out the whole tweet and operating on it.

In this example, a functor is a function that does a transformation on data inside a context. In this example the context is an Iterable and its contents are Tweets. The transformation is a functor from Tweet -> ID. This way, an Iterable<Tweet> can give me back an ID, exactly what I wanted in the first place, without me ever having to see its Tweet.

Look! A functional trick can make my Java more OO than ever.

Caveat: there are many definitions for Functor out there, and different types of functors. This is one.

Friday, September 21, 2012

When is code data, and when is it code?

The first tenet of functional programming: code is data. Store it in variables, pass it as parameters, return it from methods.

This is challenging in the JVM, where all code lives inside methods on objects. Therefore, Scala passes around code in a function object. Code lives in the apply() method.

When is a function object created? It is not always clear when we're reading Scala code: what is assigned to an ordinary method on a class? what will be executed immediately? what gets wrapped in a function object for storing and passing around?
This post describes how to get from executable code to a method; how to get from executable code or a method to a function literal; and how to get from a function literal or method back to executable code. Along the way I'll mention three exceptions to the rule "a method is executed every time it is referenced," show how to pass around a catch clause, and interrogate an innocent citizen.

Since all code in Scala is in a class (even scripts get wrapped), let's start there.

Code in a class: now or later?

Expressions inside a class body are executed on instantiation as part of the default constructor. The two exceptions are methods (declared with def)and function literals. This can surprise the unwary coder.

If a block or line of code comes after val x =, then it's getting executed once, right now. If it comes after def x =, then it gets executed later, whenever the method is called.

Say we want to create a wiretap on a citizen. The citizen is mutable. The following class will initialize the val phone only once; if Sally's home phone number is updated, the wiretap will never see it. However, name is defined as a method; if Sally's name changes, her wiretap's name property will reflect this.

class Wiretap(target: Citizen) {

    val phone = { target.homePhone }

    def name = target.name
}

Scala's Principle of Uniform Access means that users of Wiretap don't have to care whether name and phone are methods or fields, they're just properties.[1] But the writers of Wiretap had better be careful!

Function Literals

You can always defer execution of code by wrapping it in a function literal. There's a straightforward way to do this, and there are sneaky ways.

Who doesn't love rockets?

rocket operator

Create functions explicitly with a rocket. You can put these into a val if you like, or pass them to methods like List.map which expect a function type.

val isLegal = (w: Wiretap) => w.toString.length > 1

(We set the bar pretty low for a wiretap to be considered legal.)

Using the ubiquitous underscore, we can make a function literal without a rocket:

val isLegal = (_:Wiretap).toString.length > 1


What function is expected as a parameter


Inside a parameter list where a function of specific parameters is expected, we can skip the parameter-type declaration. Scala will do everything it can to turn what you put in that parameter into the expected function type.

val legalWiretaps = listOfWiretaps.filter( _.toString.length > 1 )

Scala will convert a method into a function if the method is referenced where a function of the same type is expected. This is one of three exceptions to "methods are invoked every time they are mentioned."

object Wiretap {
   def isLegal(w : Wiretap) = w.toString.length > 1
}

listOfWiretaps.filter(Wiretap.isLegal)

Because filter expects a function of type Wiretap => Boolean, Scala recognizes that the method matches this signature and wraps the method in a function literal for you.

When you want a method treated as a function literal


You can tell Scala to treat the method as a function type -- code-as-data rather than code to be executed -- if you follow the method name with an underscore.[2] The underscore triggers the Scala compiler to wrap the method instead of invoking it.

scala> val targetNameFunction = sally.name _ 
                    // the near-invisible underscore is critical
targetNameFunction: () => String = <function0>


targetNameFunction is now a function that returns string. Every time it is applied, sally.name will be called again.

This is the second exception to "every time a method is referenced, it is invoked. The last one, sneakiest of all, is described in the next section.

Where the method you call sequesters your expression


In Java, when you call a method, all the parameter expressions are evaluated before they are passed in. This is eager evaluation. In Scala, the writer of a method can specify "when they pass me something, don't bother figuring out what they passed me - just give me that code so I can evaluate it when I please." This is called (for reasons I don't understand) a by-name parameter. It's cool because it lets Scala developers define our own control structures like loops and conditionals. It's uncool because the caller of the method may not realize that she just passed in a function literal; it looks the same to her. That expression might be evaluated once, or many times, or not at all.

Here's an evil method:

   def evil(sneakyMethodThing : => String) = {
      println("Time to do some bad stuff")
      sneakyMethodThing == "I didn't do it" || sneakyMethodThing == "You can't prove anything"
   }

When I call this with an expression that interrogates Sally...

evil( officer.interrogate(sally) )

... then in this one line, Sally might get interrogated twice!

The code officer.interrogate(sally) looks to the caller of evil like an expression that will be evaluated immediately. But really it gets wrapped up inside an object and passed in to the evil method. Yet inside the evil method, sneakyMethodThing doesn't behave like a function object; you don't have to use () to apply it. It behaves more like a method: every time sneakyMethodThing is referenced, it is invoked.

evil: Time to do some bad stuff.
officer: "Why did you do it?"
sally: "I don't know what you're talking about."
[evil observes that this does not equal "I didn't do it" and continues evaluating]
officer: "Why! Why!"
sally: "I told you, I don't know!" [sobs]
[evil returns false]

Watch out for these by-name parameters, and be careful about passing in expressions (like this one) with potentially damaging side-effects.

Why should a function cover every case?


This is a strange animal. A partial function literal is one or more case statements in a block - like a chunk of a match expression. Catch expressions use these.

try {
  // some stuff
} catch {
   case ex : IllegalArgumentException => "I'm a dork"
   case ex : Throwable => "The world is over"
}

The part after the catch clause is a partial function. Check this out - you can put that piece in a value:

val standardExceptionHandling : PartialFunction[Throwable, Any] = {
   case ex : IllegalArgumentException => "I'm a dork"
   case ex : Throwable => "The world is over"
}

... and then use it:
try {
// some stuff
} catch standardExceptionHandling

That's cool!

The other place partial functions are useful is when you want to use a pattern to extract data from the parameter to a function.

someMapOfThings.foreach {
    case (key, value) => /* do something with key and value */ }

Be careful here to use a pattern that will match every input. Otherwise, a MatchError can bite you at runtime.

The PartialFunction is a function object with a bonus method: isDefinedAt returns a boolean indicating whether the function has an answer for the supplied input. These are interesting in other ways, but this post is about syntax. It's time to take this the other way and turn functions and methods back into executable code.

How to make it go


To wrap it up, go the other way: call methods and functions to execute the code inside of them.

Methods and by-name parameters are called every time they are referenced. Three exceptions: when they appear in a parameter list where a corresponding function object is expected; when they are followed by an underscore; and when they are received into a by-name parameter.

Functions are applied in two ways:
  1. follow the identifier with a parameter list. If the function accepts no arguments, then the parameter list is an empty set of parens. Unlike methods, you can't leave off the parentheses.
  2. call the apply method directly
val k = (_:String).length
  1. k("hi")
  2. k apply "hi"

There's a peculiar consequence to the first syntax here. Anything with an apply method is a function - including stuff like lists and strings and arrays.[3] If you follow a string with a parameter list, you'll get a result.

scala> "Sally"(0)
res10: Char = S

Perhaps even more startling, if you pass a string where a function is expected, it'll accept it.

scala> Range(0,5).map("Sally")
res11: scala.collection.immutable.IndexedSeq[Char] = Vector(S, a, l, l, y)

But that string is a partial function, so it'll blow up if you apply it to something out of range.

Whew


Scala juggles the code around among methods, objects, and expressions to achieve its mix of imperative and functional support. Conceptually, code can be passed as data, even on the JVM. Those concepts aren't always crystal-clear to the reader, though, so be careful.


[1] for added confusion, in a class, both methods (def) and fields (val) are implemented as methods. The val becomes a getter, essentially, that returns an invisible constant field.

[2] Scala's error messages refer to this as "partially applied function," but I dispute the use of this term. For instance, try to assign a method with parameters to a val:

scala> val e = Math.pow
<console>:8: error: missing arguments for method pow in class MathCommon;
follow this method with `_' if you want to treat it as a partially applied function

Again, saying "Math.pow _" will wrap the method in a function type and store that in the val. I think this is not partial application; @bvenners says it's eta-expansion. @dickwall says it is an edge-case of partial application, supplying 0 arguments.

[3] Or there's an implicit conversion that turns them into something that implements PartialFunction. Always tricky in Scala.

What is flatMap?

flatMap is a method on collections that, at a surface level, transforms each element into another collection of any type, then takes all the elements out of those collections and puts them into one. It is like a map followed by a flatten.

Why do two things in one step?

Conceptually, List.map is always 1:1 - each element in the input collection produces one element for the output collection. With flatMap, each element in the input collection produces 0 or more elements for the output collection.

This is like the map in CouchDB's MapReduce: the map function can emit zero or more output documents for each input.

Or as @Deech says, "It's just a monadic bind." In slightly more colloquial terms, we can think of it as: pass a functor into a context, and get back the same kind of context with different stuff in it.

Sunday, September 16, 2012

Abstraction goes both ways

Abstraction is critical to programming. It is the core activity we use to make more interesting and complex programs.

Most of us understand abstraction as finding commonalities between different concepts, and modeling these as an inheritance hierarchy. If you are a retail store, then toy cars, trash cans, and pillows are all sellable items.  Thus, abstraction is finding similarities in disparate things. Identifying patterns.

But there is another side: breaking concepts down into components. While the trash cans and toy cars are the same at the cash register, they are different when stocked on shelves. That GUI button has a view, a model, and some event triggers. Thus, abstraction is also finding differences in what appears to be same. Breaking what appears to be an indivisible whole into disparate components. Design is the process of breaking things apart.

This second application of abstraction is important in analyzing cause and effect. We started an agile process, and then a bug made it into production. Agile is a failure! But really, did our process changes have anything to do with it? Perhaps the two events were unrelated. Or perhaps the same frustrations led to both. As humans we get overexcited about X happened, and then Y happened; therefore X led to Y. We are too quick to form patterns where none exit. Laurie won both poker nights, therefore Laurie is skilled at poker. It is challenging to break Laurie's play into components of luck and skill - skill we can control, so it's a much more appealing cause. Breaking a cause or effect into multiple components requires abstraction, and if we use it, we will be better at programming and at life.

Tuesday, September 11, 2012

Iterators in Scala: glory and danger

In the beginning, there were Iterables. And in the old world of Java, they were good. We could perform the advanced for loop on them.

for (Thing item : listOfThings) {
   // do stuff to each item
}

And we could perform it again and again.

for (Thing item : listOfThings) {
   // do more stuff to each item
}

but what is this Iterable? What requirement must be met by the protagonist in the advanced for loop?

interface Iterable[T] {
    Iterator[T] iterator()
}

What is this inner concept, this offspring of the Iterable? Is this Iterator of the same species, and can we impose upon it in the same manner?

In Java, clearly no.

for (Thing item: listOfThings.iterator()) { 
             // compile error: foreach not applicable
}

Yet, an Iterator can be useful to us. It can provide the contents of a file - scala.io.Source in Scala  - or draw from any data source local or remote.

In Scala, more possibilities are open to us. We can use the Iterator in a for-expression. We can map over it, filter it, and many other operations that apply equally to Iterable.

val lineIterator = Source.fromFile("build.sbt").getLines

for( line <- lineIterator) yield {
   // something interesting about each line
}

Yet! Beware! Scala tempts us to use Iterators interchangeably with Iterables. There is danger here. We are spoiled by traversing Iterables such as lists.

val usefulThings = listOfThings.filter(isUseful(_)) // great

val blueThings = listOfThings.filter(isBlue(_)) // great

Most of the collections we use in Scala are immutable and stateless. We can traverse them all day; they continue to supply iterator after iterator upon request. Lists and Sequences implement the trait Traversable.
With iterators, beware! Once accessed, they are forever altered. Iterators are inherently stateful. Every call to next() yields a different result. They implement the trait TraversableOnce.

val lineIterator = Source.fromFile("build.sbt").getLines

val usefulLines = lineIterator.filter(isUseful(_)) // danger!

We now hold two references to the lineIterator. One is contained inside usefulLines, itself an iterator that holds a reference to lineIterator. Any access to usefulLines - even calling toString, which prints whether it is empty! - will change the state of lineIterator. Once you apply any operation to an Iterator, its internal state is unpredictable. Throw away its reference. The Iterator is TraversableOnce.

The immutability of Lists and Maps and Vectors can spoil us. We must remember to track a val which identifies a mutable object such as an Iterator.
With such danger lurking, why should we use an Iterator in Scala?

Sometimes state is needed. Since Iterators are inherently stateful, they are a reasonable place to encapsulate state.
One day, not so long ago, I wished to access the Twitter search API. The tweets were to be retrieved repeatedly, each time excluding tweets already retrieved. The "since" parameter to the Twitter API accomplishes this passed the ID of the last tweet already retrieved. This ID is state. This state can be encapsulated in an iterator.


class TwitterIterator extends Iterator[Tweet] {
      var lastTweetReceived = "0" // mutable state
      
      def hasNext = true // the twitter stream has no end

      def next() = {
        val tweet = retrieveNextTweet(lastTweetReceived)
        lastTweetReceived = tweet.id
        tweet
      }
}


Two other problems which barred my path in the last fortnight yielded to this same solution: encapsulate the state in a custom iterator. Thus, Iterators are an idiom of usefulness. And of caution! In Scala, Iterators appear to be interchangeable with Iterables, but they most decidedly are not. Once an iterator is given to any function or expression, it shall be dead to you. Access it not except through the result of the single expression in which it appears. Store it not in a val, lest you or your compatriots attempt to access it and render it forever into an unknowable state.

def in Scala: equals sign or no equals sign?

When creating a method or procedure in Scala, we start with def, then the name and parameters (if any), optionally the return type, and then the method body. There's a subtlety here: sometimes the method is equal to the method body, and other times the equals sign is missing. For instance:

class Birthday {
   def howManyMoreDays(from : Date) = {
       // perform some calculation
       daysLeft // return last expression
   }
   def getExcited() {
 
       println("Oh yeah! It's your birthday!")
       "no one's listening."
   }
}

In this case, howManyMoreDays uses an equals sign between the declaration and the method body. getExcited does not. What is the difference?

Methods without an equals sign always return Unit. Methods with an equals sign between the declaration and the body always return something: the result of the last expression in the method body.

Without an equals sign, you have a procedure, performed for its side effects. With the equals sign, you have a function, performed for its result. Use them accordingly; avoid side effects in functions. The procedure, marked by its lack of an equals sign and its useless return value, is a subtle way to call out side effects, to make it them more explicit.

You cannot return anything but Unit from a method without an equals sign between the declaration and the body. Adding any other return-value declaration is invalid. The last expression here is not returned; no one ever sees the string "no one's listening."


If we forget the equals sign in the definition of howManyMoreDays, it will not return anything. To avoid this pitfall, consider explicitly declaring the return type. The following will give a compile error on the method definition, instead of waiting until we use the method:

   def howManyMoreDays(from : Date) : Int // error! need =
       // perform some calculation
       daysLeft // return last expression
   }

For more subtleties in method declarations in Scala, consider def vs val and empty parens vs none.

Sunday, September 9, 2012

Git: The Good Parts - history is written by the victors

Why change history?

Version control has two reasons for being: save your work, and tell the story of your project. These two goals are in conflict. The more often you save your work, the more jumbled and cluttered the story of your project.

"Time for lunch" is a great reason to save your work, but it makes a lousy commit message for posterity.

Git solves this: it lets you rewrite history. It says, "Do what you need to do, and then decide how you want to remember it." Why take this extra step? Because software is not written file by file, line by line. Software is built feature by feature, fix by fix. This is what you want to see when you look back at history.

How to change history

There are two aspects to telling your story in git: rewriting feature branch history and bringing feature branches into mainline. This post covers the former.

Before I share a change with other people, I use interactive rebase to put the changes in a rational order. It's like going back to a savepoint in a video game to optimize my play, except I don't have to fight the enemies again; I just tell git what order I should have fought them, and git replays the game in that order.

Let's pretend that I slayed a troll, an ogre, an innocent bystander, the big boss, and then a stray ogre. I want it to look like I killed two ogres at once, then the troll, then the big boss; leave innocent bystander alone.
Currently, my commit history looks like this:
The first step is to find that savepoint. Often this is where our work diverged from origin. The farthest you can go back is the last commit that was fetched or pushed. Never change shared history. (git will let you change it - Bad Part Alert.) Take a look at the commit graph in gitx or gitk, or use git log, and find the last good commit that you don't want to change. In this example, it's the green commit where I entered the level.

Next, take a deep breath and type:
git rebase --interactive b6f5708
Put the hash of your commit in place of mine, of course. (Bad Parts Alert: if you use a branch name instead of a commit hash, different things happen.)

*Poof* some magic happens and a file opens before your eyes. Each post-savepoint commit is listed, oldest first. Part of the commit hash and the first line of the commit message identifies each. Each is prefixed by the mysterious word "pick."
pick a4546fe killed a troll
pick 07c4384 killed an ogre
pick c8370ba killed an innocent bystander
pick 4bc809a killed the big boos
pick f07833b killed one more ogre
The file also contains comments containing handy instructions. The concept is: mess with the contents of the file, replacing pick with one of the other spells, until the list of commits looks like what we wish we had done. The good parts here are pick and squash. Pick means "use this commit" and squash means "use these changes, but put them into the previous commit."

There are two less-obvious spells: the order of lines in the file controls the final order of commits, and deleting a line from a file eliminates that commit from the final result. Be careful with that [1].

Oh hey! While I'm in here, I notice a typo in the commit message where I killed the big boos. Changing it in this file has no effect, but changing pick to reword will trigger git to open the commit message for editing.

Change the file by rearranging the lines to the correct order, and making the second ogre commit say "squash":
pick 07c4384 killed an ogre
squash f07833b killed one more ogre
pick a4546fe killed a troll
reword 4bc809a killed the big boos
With much anticipation, I save and quit. I am stymied by another editor window: the squashed-together commit wants a new commit message. In this case, I'll record this as "killed two ogres," then save and quit again. Next the commit message I want to reword opens; fix this, save and quit.

Finally, the rebase is complete. Our old commits go away, and new ones are created in the order specified. our history looks like this:

Hurray! Our branch is all pretty.
Real life happens on the unhappy path
Rebase doesn't always go this smoothly. As the rebase is recreating each commit, conflicts are possible. For instance, two commits might change the same part of a file, and the rebase re-orders them. When this happens, git stops the rebase and asks you to resolve the conflicts. You have two options:
  • use git status to see the files with conflicts; edit them to resolve the conflicts; use git add <file> to tell git they're ready to go; git rebase --continue.
  • abort! abort! git rebase --abort and try something different. This gives up on the entire rebase and puts you back where you started. [2]
Bad Part Alert: until the rebase is either aborted or continued, your repository is in a strange state, and weird things could happen.
From here, it's time to get these changes into the mainline branch - the different ways to do that is a topic for another post.

If you thought this was fun, there are other reasonable ways to rewrite history: change commit messages; split one commit into multiple commits; move some commits to a different branch. These are stories for another day. For now, keep saving your work, and consciously change your history so your coworkers don't know what time you went to lunch.

[1] deleting a line from the rebase file is not entirely a good part - the functionality is useful, but this is a cruel way to implement it. I wish we could change "pick" to "skip" instead, and receive an error if we accidentally delete a line from the file.

[2] if you find yourself restarting rebases or merges, you might want to enable git rerere, because it saves you from repeating the same merges.

Monday, September 3, 2012

Git: The Good Parts - moving files between trees

Git is the best version control system available today, miles ahead of centralized VCS. It has beautiful, consistent concepts internally. But its API is crap. Commands and options are inconsistent, internal details are exposed, and there are many ways to perform the same operation. Let's try to narrow the options to only the ones that are a good idea.

Learning git command-by-command is difficult, because each command has multiple jobs. (I'm looking at you, git reset.) The objective of this post and later posts in this series is to learn the best commands and forms for advisable operations, trimming away the hairy details. Like every other language and tool that's too flexible for our own good, git will serve us better if we stick to the good parts.

Let's look at what Scott Chacon calls the Three Trees in git. These are the three locations where files are stored: the working directory, the staging area, and in a commit (inside the repository). I want to see the best way to find out what is different between these trees, and how to move files around between them.

The first level of simplification is settling on consistent terms for the three trees. The help pages are not consistent, but I will be.

Working directory: this is your project directory on the local filesystem.
Staging area: the nebulous half-formed commit that will be next.
HEAD commit: the most recent commit on your current branch. Each commit contains a tree representing the state of all the files in the repository.

Find out what's different between them:
The git diff command by default shows the difference between the working directory and the staging area - these are listed in git status under "Changes not staged for commit." If you want to see all the differences between your working directory and the last commit, use git diff HEAD.

Next, move files between these three trees:
Caution: these commands (other than commit) need a path argument. You can use . as the path to say "all the files," but leaving off the path for git reset or git checkout can cause the command to do something entirely different. Specifically, if you provide a different commit as an argument instead of the default of HEAD, scary stuff happens. If your objective is only to modify files in the working directory or staging area, be sure to specify a path.
Of course, git commit doesn't put the files in the staging area into the current HEAD commit. Rather, it creates a new commit with that tree, and makes that new commit the HEAD. This is not pictured.

There are many other ways to see differences and move files between these trees, including other commits besides HEAD. For the full range of options, check out the amazing git cheatsheet.

Look for more Git: The Good Parts posts right here, on your friendly neighborhood blogitron.