Open Source needs Open Source companies.

The other day, AWS announced its latest plans to work around the license of ElasticSearch, a very useful open source project cared for by Elastic.

“At AWS, we believe that maintainers of an open source project have a responsibility to ensure that the primary open source distribution… does not advantage any one company over another. “

Can I call BS on this?

I mean, I’m sure AWS does believe that, because it’s in their interest. But not because it’s true.

Great pieces of software are not made of code alone. They’re made of code, which is alive in the mental models of people, who deeply understand this software and its place in the wider world. These people can change the software, keep it working, keep it safe, keep it relevant, and grow it as the growing world needs.

These people aren’t part-time, volunteer hobbyists. They’re dedicated to this project, not to a megacorporation. And they eat, and they have healthcare.

Reusable software is not made of code alone. It includes careful APIs, with documentation, and of communication lines. The wider world has to hear about the software, understand its purpose, and then contribute feedback about problems and needs. These essential lines of communication are known as marketing, developer relations, sales, and customer service.

Useful software comes out of a healthy sociotechnical system. It comes out of a company.

If a scrappy, dedicated company like Elastic has a license that leans customers toward paying for the services that keep ElasticSearch growing, secure, and relevant — great! This benefits everyone.

Poor AWS, complaining that it doesn’t have quite enough advantages over smaller, specialized software companies.

Then the next sentence: “This was part of the promise the maintainer made when they gained developers’ trust to adopt the software.” Leaving aside the BS that a person who gives away software owes anyone anything —

The best thing a maintainer can do for the future of all the adopters is to keep the software healthy. That means watching over the sociotechnical system, and that means building a company around it. A giant company fighting against that is not on the side of open source.

The Emperor has no clothes: Bad actors in tech

Maybe you are interested in a language, or an open source project, but you feel like the community is unwelcoming: Some big voices are rude, they’re downright hostile to newcomers and anyone who disagrees with them. Let’s not get involved.

Or in your workplace: influential people in the organization aren’t nearly as helpful, or as smart, as your teammates. Yet their opinions, and bad behavior, are followed by everyone else, and you feel bullied into decisions, you accept their little abuses. Ultimately the people with the most freedom move away, until the workplace is a surreal island where time stood still

There’s this old Christian Andersen tale, The Emperor’s New Clothes. Some weavers sold the emperor a suit, which cannot be seen by those who are incompetent or unfit for their position. It was really not a suit: they gave him nothing, the emperor was running around nude. Nobody said a thing. Each assumed they were in the wrong. Only when a child cried out “he has no clothes!” did everyone else realize that they were not alone in pretending.

While the Emperor’s situation is humorous and extreme, the open source and workplace situations are not. They are real, and they have mathematical underpinnings! A paper, called “The Majority Illusion in Social Networks,” studies this very phenomena. The Washington Post describes how public opinion appears to change rapidly, when sometimes it’s really that public opinion suddenly became known.

Those whose opinions are the most visible control what opinions are seen as acceptable and polite. The moment enough of the social network sees their own opinion as acceptable, bam! a major change in sentiment appears. That feeling was already there, but it was masked by a few local celebrities who didn’t share the values of the majority.

When we dislike bad behavior, we feel alone, but we are not special. We all see it, we all dislike it. We all wish we were using the right tools for the job. We all wish the mailing list contained only polite helpful responses. But social norms — set by the few who are also the loudest — make us not care enough, not invest enough, to communicate our opinions with such volume. Shame and fear of rejection freeze us. Or we lack the hunger for conflict. None of us say the emperor has no clothes, bad behavior persists.

The math says the evident culture is not always the predominant culture. A few confident, inconsiderate people in key positions intimidate an entire department. A few derogatory voices fence off a language, leaving erstwhile contributors to chew on rocks outside.

If you care about your organization, work to make everyone’s voice heard.
If you care about your open source community, speak out against behavior that masks the majority attitude. Watch for a red flag in your head: “I think his opinion is wrong, but I’m not going to say anything because arguing with him is not worth it.” You’re not the only one who sees through that clothing.

Software is a tree

Maybe software is like a tree.

The applications, websites, games that people use are the leaves. They’re the important part of the tree, because they’re useful. They solve real problems and improve lives, they make money, they entertain.

The leaves are built upon the wood of the tree, branches and trunk: all the libraries and languages and operating systems and network infrastructure. Technologies upon technologies, upon which we build applications. They’re the important part of the tree because dozens to thousands of leaves depend on each piece of wood.

Which part of the software tree do you work on? Is it a leaf? Leaves need polished according to their purpose, monitored according to their criticality, grown according to users’ real needs. Is it the wood? The wood needs to be strong, very well-tested because it will be used in ways its authors did not imagine.

Ever find yourself in between, changing an internal library that isn’t tested and documented like a strong open-source library, but isn’t specific to one application either? I’ve struggled with code like this; we want to re-use it but when we write it we’re trying to serve a purpose – grow the leaf – so we don’t stop to write property tests, handle edge cases, make the API clear and flexible. This code falls in the uncanny valley between wood and leaf. I don’t like it.

Code that is reusable but not core to the business function is best shared. Use another library, publish your own, or else stagnate in the uncanny valley. Follow the mainstream, form a new main
stream, or get stuck in a backwater.

With a few days set aside to focus on it, we could move that shared code into the wood, and release it as open source. Make it public, then document, test, and design with care. When we graft our twig into the larger tree, it can take nourishment from the contributing community, and maybe help more leaves sprout.

If the code isn’t useful enough to release as even a tiny open-source library, then it isn’t worth re-using. Build it right into multiple leaves if multiple leaves need it. App code is open for modification, instead of the closed-for-modification-but-needs-modification of the shared internal library.

With care, ours can be a healthy tree: millions of shiny custom leaves attached to strong shared branches.

JVM Threads, ???, and Open Source

Lately, open source is important to me. Today, for instance: it starts with a useful library, complicated by Java threading, confused by a Scala language feature, and ends with happiness and joy because Open Source.

I’m using scalaz.concurrent.Task, a monad (LINK) with at least three special powers. First, it use a Future internally, which perform asynchronous computation. Second, it uses scalaz.concurrent.Future (as opposed to the built-in one) because this one trampolines from thread to thread so that we never run out of stack no matter how many “and then do this” computations we hook up before running it. Finally, Task handles errors, wrapping all its work in a try/catch and retaining exceptions in scalaz’s version of Either (called \/ for \/ictory).

I can build up a Task starting with one delayed computation, adding on transformations, and then binding on transformations that produce other Tasks, as many as I like. (See the functor and monad LINK posts for context of the drawings.)

Task wraps each calculation in a try/catch, translating exceptions into data, so they can get passed back to whoever evaluated the Task.

Then, scalaz.concurrent.Future tosses each transformation over to another thread. Only the original thread blocks; after that the calculation hops from thread to thread until it’s all executed (or failed), and then an answer returns to the waiting thread.

This is all a giant game of “You can’t catch me” with StackOverflowError. It works, until…

a train car blows up in one of the threads. If that caution tape isn’t wound tight enough, and the code throws an exception and nothing catches it, well – that thread just keels over.

The blocking thread waits, and waits, and waits… the program does not terminate. This is how Java threads work. An uncaught Throwable doesn’t destroy the process, only the one thread.
I hate it when that happens! and it kept happening to me!

In this case, my code was deliberately throwing an exception. I was testing exception handling. But…

object format: JsonFormat[BadGuy] = {
   def read = ???
   def write = throw new RuntimeException(“poo”)

unbeknownst to me, my code called read before write. Here comes the other tricky bit: ??? is Scala for “throw NotImplementedError.” Which is often useful; it’s basically “WTF you weren’t supposed to call this yet.”

??? throws an Error, not an Exception. Both are subclasses of Throwable. In general, it’s bad form to catch Throwable, because Errors are not intended to be anything you can recover from, like OutOfMemory or StackOverflow. I guess one doesn’t normally recover from unimplemented methods, so it’s an Error.

Task followed this rule of thumb: it caught Exception but not Throwable. That’s how ??? caused thread death; that’s why my test never terminated. It kept chewing on that Task forever.

What to do? I think the library is wrong. Sure you’re not supposed to catch Throwable most of the time, but this seems like an exception. If we want Error to terminate the program, the Task should catch it, pass it back to the original thread, and throw it there. How can I ever be sure my programs won’t die a painful slow undeath of starvation?

Open Source to the rescue! I can do more than kick my desk, more than gripe to my coworkers. More than complain on twitter or here. First, I posted the question to the mailing list. Scalaz’s Task was written by Paul Chuisano, and he responded superfast. Not only did I get an answer, I got to fix it!

Instead of bemoaning my fate, I have the warm fuzzies and pride of contributing to a library that I care about. Future users of Task get to miss out on this particular source of undying, unmoving processes. Everyone wins!

So the next time your program terminates, say a prayer of thanks open source software.