Rebase on the World

We build our software in a particular world, a world of technologies that we link together. We choose a programming system (language, runtime, framework), libraries, and environment. We integrate components: databases, logging, and many different services.

Perhaps we built it on Java 8 running on VMs in our datacenter, connecting to a proprietary queuing service we bought years ago. We start with what is available and stable at the time.

But do we stay there?

The outside world moves capabilities toward commodity.

At some point, new businesses start building cloud applications instead of racking their servers.

An opportunity appears, and our enterprise can get out of the infrastructure business. When we shift our application onto AWS, there are whole areas of expertise we don’t need in-house. There are layers of infrastructure that Amazon maintains and upgrades, and we rarely even notice.

At some point, we integrate with new systems. They don’t speak our proprietary queuing protocol, so we move to Kafka, something that people and programs everywhere can understand. And at some point, new businesses don’t run Kafka; they rent it as a service.

When we move to SaaS, there’s a layer of expertise we don’t need to retain, pages we don’t have to answer, and upgrades we don’t have to manage. Or even better, maybe our needs have changed, or SQS has improved until it’s good enough. We get free integration with other AWS services and billing.

Is our software simpler? I don’t know, but it’s thinner. The layer we maintain is closer to the business logic, with integration code to link in SaaS solutions that other companies support.

All code is technical debt.

Every line of code written is in a context. Those contexts change, and expectations rise. New tools appear, and integrating them gives us unique abilities. Security vulnerabilities go noticed.

For the software we operate, we are responsible for upgrades. It is our job to keep libraries up to date, shift to modern infrastructure every few years, and add the features that everyone now expects.

What you get for operating custom software — you control the pace of change.  
What you pay  — you are responsible for the pace of change.

Maybe it’s authorization, or network configuration, or caching, or eventing. You wrote it back when your needs were exceptional, and now it’s your baby, and you’re changing its diapers. It takes effort to shift to anything else.

Incorporate the modern world into our software’s world.

When capabilities become commodities, it becomes cheaper to rent than to babysit them. It’s probably monetarily less expensive, and indeed, it’s less costly in knowledge. People and teams are limited by how much experience we can hold. We can only have current expertise in so many things.

On a development team, we can increase our impact by overseeing more and more business capabilities, but we can only operate so much software. If we thin that software by shifting our underpinnings to SaaS offerings, then we can keep up more of the software that matters to our particular business.

All code is technical debt. Let it be someone else’s technical debt. Move it off your balance sheet, to a company that specializes in this capability.

Rebase on the world

In git, sometimes I add some features in a branch, while other people improve the production branch. When I rebase, I put my changes on top of theirs, and remove any duplicate changes.

I want to do this with software infrastructure and capabilities. The outside world is the production branch. When I rebase my custom software on top of it, it takes work to reconcile similar capabilities. But it’s worth it.

When we rebase our software on the world, we get everything the world has improved since we started, we get integrations into other systems and tools, and we get learnings from experts in those capabilities. SaaS, in particular, has a bonus — we keep getting these things, for no extra work!

If we don’t rebase on the world, a startup will.

How can a scrappy little company defeat a powerful incumbent?

Every piece of software and infrastructure that the big company called a capital investment, that they value because they put money into it, that they keep using because it still technically works — all of this weight slows them down.

A startup builds on the latest that the whole world offers. They write minimum code on top of that to serve their customers. The less code they have, the faster they can change it.

In the 1990s, we built a big stack of custom work on top of a solid base. In the 2010s, we build less custom software to get the same business capabilities (with more reliability) because we’re building on various AWS services and many other tools and services.

This is not the only advantage a startup has, but it is a big one.

Software is never “done.”

Software is not bought, it is rented. (Regardless of how the accounting works.) It gives us capabilities as long as it keeps running, keeps meeting expectations, keeps fitting in with other elements of the world that need to integrate with it.

Keep evolving the software, infrastructure, and architecture. It is never going to be perfect, but we can keep it moving.

When I’m coding a feature, I rebase on the production branch every few hours. For software systems, try to rebase on the world every few months, bit by bit.

In an enterprise with a lot of code, this is an extra challenge. Change at that scale is always an evolution.

If you find yourself thinking, “we have so much code. How could we ever bring it all up to date?” then please check out Atomist’s Drift Management. Get visibility into what you have, and even automatic rebasing (of code, at least). There’s a service for this too.

Acknowledgment
A large amount of this information came out of a conversation with Zack Kanter, CEO of Stedi.

Migrating some services from AWS to Pivotal Web Services

My objective is to run some services on Pivotal Web Services (PWS; hosted instance of Pivotal Cloud Foundry), and have them respond to requests to `https://survey.atomist.com` at various paths. Currently these services run on AWS, along with services that respond at other subdomains of atomist.com.

TL;DR: this is easy enough for HTTP requests and prohibitively difficult for real HTTPS, for only one subdomain.

This posts describes some tricky bits in this process, and the bits that leave me stuck.

Prerequisites: I have PWS set up and a few apps deployed. Meanwhile all our existing infrastructure runs on AWS.

First: multiple apps responding at satellite-of-love.cfapps.io

The instructions tell me how to point my own domain at a single app in PWS, but I want multiple apps to serve paths from my domain. The caller should not know or care which service is responding to its request for a resource.

To do this, I set up a route in cloud foundry, with a hostname (which seems to be PCF’s name for the third-from-the-right segment of the domain name, anyone know why?) that doesn’t correspond to any one app.

`cf create-route jessitron cfapps.io –hostname satellite-of-love`

Here, jessitron is my space in PWS. cfapps.io is PWS’s domain, this gets requests into Cloud Foundry for routing. satellite-of-love is a domain name that I like, it matches my github org.

That path is going to 404, but I have called dibs on satellite-of-love.cfapps.io. It’ll route to my jessitron space and no one else’s.

Now I can make routes for each endpoint and tell it which app serves it. For the /vote endpoint on Kitty Survey, I have an app running called london, so I hook that up:

`cf map-route london satellite-of-love.cfapps.io –path /vote`

Now I can hit https://satellite-of-love.cfapps.io/vote and my london app receives a request at path /vote. This is good for testing.

This part totally works with HTTPS. If you don’t mind changing your clients to point to this URL, stop here.

Second: HTTP: pointing survey.atomist.com to satellite-of-love.cfapps.io

This is DNS setup. We happen to use AWS Route53 for this. I go into the AWS console to set up a CNAME record for survey.atomist.com -> satellite-of-love.cfapps.io. There was one tricky bit to this in Route53: I clicked on the existing survey.atomist.com record (if it didn’t exist I would click Create Record Set), and tried to enter my target BUT NO
It was all “The record set could not be saved because:
– Alias Target contains an invalid value.

Here’s the trick: choose Alias: No.

With a regular CNAME (the Alias ones are an internal-to-AWS thing), I can route to an external domain from Route53.

Next, over in Cloud Foundry land, I can tell it about this domain.

 `cf create-domain atomist survey.atomist.com`

Here, atomist is my PWS org. Then I tell it to send requests to my space please:

`cf create-route jessitron survey.atomist.com`

And then I create routes for each of the endpoints, but with this new domain. (I’m pretty sure this is necessary.)

`cf map-route london survey.atomist.com –path /vote`

I’ll need to make these two routes (or at least the last one) for every endpoint I add to my service. Soon I’ll add this to my “add REST endpoint” automation in Rug.

Third: security certificates and https

There are two ways to get an HTTPS:// endpoint on PWS. They recommend using CloudFlare, which can be free. There are two problems with that. 

CloudFlare -> Cloud Foundry

The first is, to route anything at atomist.com through CloudFlare, I have to route atomist.com through CloudFlare. I have to change the routing for my entire company. 🙁
Even if I did reroute our whole domain through CloudFlare, the second problem appears: I can get the appearance of security but not actual end-to-end SSL. The easy option to choose is “Flexible”, meaning users get SSL from browserCloudFlare and it looks secure to them, but behind the scenes it’s HTTP between CloudFlare and my app. This seems unprofessional to me, letting everyone’s requests happen without SSL behind the scenes while telling them it’s secure.
The other option to choose is “Full SSL,” but then I need SSL on Cloud Foundry anyway, so …

SSL in Cloud Foundry 

There’s a Pivotal SSL service available in the PWS marketplace for SSL termination. For $20/month (they don’t mention that in the documentation), it’ll let you upload one certificate.

Currently, we use AWS Certificate Manager, which provides free certificates that only work on AWS.
Can I get that for survey.atomist.com separately, while leaving the rest of atomist.com alone? I’m going to try that, from some other source — but not today.

Therefore, because our security certificates are tied to AWS
and because I decline to change the routing of our entire domain in order to experiment with this subdomain,
I give up. My toy website doesn’t need HTTPS anyway.
The moral is: if you want to experiment with moving part of your infrastructure off of AWS (designated by a subdomain), be prepared to change how requests are routed to the root domain.
Thank you to Glenn Oppegard for information about SSL on PWS, and Simon Brown who is finding this just fine with CloudFlare.