Tuesday, July 18, 2017

Dictionary Objects in JavaScript and TypeScript

TL;DR: when using an object as a dictionary in TypeScript/ES6, iterate through it using `Object.keys()`.

Coming from statically typed languages, I keep looking for a Map or Dict type in TypeScript (or JavaScript). People use objects for this, though. Objects have key-value pairs in them, and you can add them and delete them and declare them statically and it looks nice, especially in literals.

const dictionary = { 
   impute: "attribute to",
   weft: "the holding-still strings on a loom",
}

But using a JavaScript object as a dictionary has its quirks.

There are three ways to iterate through the values in the object. (This is TypeScript syntax.)

Flip through the enumerable keys defined on an object:

for (const key of Object.keys(dictionary)) {
   const valuedictionary[key]
   console.log(`${key} -> ${value}`)
}

This one makes the most sense to me; it's what I expect from a dictionary. It flips through the values in an array of the keys defined on that object directly, no prototypal inheritance considered. This is how JSON.stringify() prints your object.

Flip through the enumerable keys defined on that object and its prototype chain:

for (const key in dictionary{
   const value = dictionary[key]
   console.log(`${key} -> ${value}`)
}

This is the easiest one to write. It flips through the keys defined on the object and its prototype chain. If you're using an ordinary object for this, and no one has done anything bizarre like add an enumerable property to Object, it's fine. tslint hates it though; it bugs me about "for(...in...) statements must be filtered with an if statement." tslint is like "OMG, you do not know what is on that thing's prototype chain it could be ANYTHING"

I find it backwards that for(...in...) flips through property names of an object, while for(...of...) flips through values in an array. This confuses me daily in TypeScript. If you accidentally use for(...of...) instead of for(...in...) on an object, then you'll see 0 iterations of your loop. Very sneaky. If you accidentally use for(...in...) on an array, you get the indices instead of the values, also confusing. TypeScript and tslint don't warn about either error, as far as I can tell. :-(

Flip through the enumerable and non-enumerable keys defined on that object:

for (const key of Object.getOwnPropertyNames(dictionary)) {
   const value = dictionary[key]
   console.log(`${key} -> ${value}`)
}

This one flips through only keys on the object, not its prototype chain, and also gives you the names of non-enumerable properties. You probably don't want those.

What are non-enumerable properties?

Conceptually, they're properties that don't make sense to flip through, that we don't want JSON.stringify() to look at. They're hidden from for(...in...) iteration and from Object.keys(). You can still access them on the object. For instance, constructors of TypeScript classes are non-enumerable properties. Methods on built-in types like Array and object are non-enumerable. They stay out of the way.

When would we want to flip through them, like in Object.getOwnPropertyNames()?
I don't know, maybe for debugging.

Why make a non-enumerable property?
I hit a use case for this today: serializing an instance of a class with recursive fields. JSON.stringify() can't print recursive structures.

Side quest: Making recursive objects printable

In TypeScript, every field in a class instance (including any inherited field) is an enumerable property, returned from Object.keys() and printed by JSON.stringify(). See this TreeNode class that tracks its children, and its children track it:

class TreeNode {


private _parent: TreeNode;
public children: TreeNode[] = [];


public constructor(public readonly value: string) {
}
...
}

Printing an instance of TreeNode gives me: `TypeError: Converting circular structure to JSON` waah :-(

Here's a tricky way to say "Hey JavaScript, don't print that _parent field". Explicitly override its enumerable-ness in the constructor.


class
TreeNode {

private _parent: TreeNode;
public children: TreeNode[] = [];

public constructor(public readonly value: string) {
Object.defineProperty(this, "_parent", { enumerable: false });
}
...
}


We can get tricky with TypeScript class fields. After all, they get tricky with us.

Properties on a class instance

In TypeScript, class instance fields show up in Object.keys() and accessor-based properties (like with getters and setters) do not. They're properties on the prototype, which is the class itself. So if you want to see accessor properties, use for(...in...) on class instances. That gets the enumerable properties from the prototype chain. Watch out: these include methods.

Why iterate through the properties on a class? I don't know, maybe for debugging again. If you do it, I suggest skipping methods. This makes tslint happy because its an if statement:

for (const propertyName in classInstance) {
  if (typeof classInstance[propertyName] !== "function") {
    console.log(`${propertyName}=${classInstance[propertyName]}`);
  }
}

Recommendations

If you have a class instance, access its properties with dot notation, like treeNode.children. That way TypeScript can help you avoid mistakes. If you have a dictionary object, access its properties with index notation, like dictionary["impute"] (and turn off the angry tslint rule). Class instances have specific types; dictionary objects are type object. Access the contents of a dictionary using Object.keys().

Friday, July 14, 2017

Migrating some services from AWS to Pivotal Web Services

My objective is to run some services on Pivotal Web Services (PWS; hosted instance of Pivotal Cloud Foundry), and have them respond to requests to `https://survey.atomist.com` at various paths. Currently these services run on AWS, along with services that respond at other subdomains of atomist.com.

TL;DR: this is easy enough for HTTP requests and prohibitively difficult for real HTTPS, for only one subdomain.

This posts describes some tricky bits in this process, and the bits that leave me stuck.

Prerequisites: I have PWS set up and a few apps deployed. Meanwhile all our existing infrastructure runs on AWS.

First: multiple apps responding at satellite-of-love.cfapps.io


The instructions tell me how to point my own domain at a single app in PWS, but I want multiple apps to serve paths from my domain. The caller should not know or care which service is responding to its request for a resource.

To do this, I set up a route in cloud foundry, with a hostname (which seems to be PCF's name for the third-from-the-right segment of the domain name, anyone know why?) that doesn't correspond to any one app.

`cf create-route jessitron cfapps.io --hostname satellite-of-love`

Here, jessitron is my space in PWS. cfapps.io is PWS's domain, this gets requests into Cloud Foundry for routing. satellite-of-love is a domain name that I like, it matches my github org.

That path is going to 404, but I have called dibs on satellite-of-love.cfapps.io. It'll route to my jessitron space and no one else's.

Now I can make routes for each endpoint and tell it which app serves it. For the /vote endpoint on Kitty Survey, I have an app running called london, so I hook that up:

`cf map-route london satellite-of-love.cfapps.io --path /vote`

Now I can hit https://satellite-of-love.cfapps.io/vote and my london app receives a request at path /vote. This is good for testing.

This part totally works with HTTPS. If you don't mind changing your clients to point to this URL, stop here.

Second: HTTP: pointing survey.atomist.com to satellite-of-love.cfapps.io


This is DNS setup. We happen to use AWS Route53 for this. I go into the AWS console to set up a CNAME record for survey.atomist.com -> satellite-of-love.cfapps.io. There was one tricky bit to this in Route53: I clicked on the existing survey.atomist.com record (if it didn't exist I would click Create Record Set), and tried to enter my target BUT NO
It was all "The record set could not be saved because:
- Alias Target contains an invalid value."

Here's the trick: choose Alias: No.
Alias: No is the "Do Things Right Button" for CNAME records in Route53 that point externally
With a regular CNAME (the Alias ones are an internal-to-AWS thing), I can route to an external domain from Route53.

Next, over in Cloud Foundry land, I can tell it about this domain.

 `cf create-domain atomist survey.atomist.com`

Here, atomist is my PWS org. Then I tell it to send requests to my space please:

`cf create-route jessitron survey.atomist.com`

And then I create routes for each of the endpoints, but with this new domain. (I'm pretty sure this is necessary.)

`cf map-route london survey.atomist.com --path /vote`

I'll need to make these two routes (or at least the last one) for every endpoint I add to my service. Soon I'll add this to my "add REST endpoint" automation in Rug.

Third: security certificates and https

There are two ways to get an HTTPS:// endpoint on PWS. They recommend using CloudFlare, which can be free. There are two problems with that. 

CloudFlare -> Cloud Foundry

The first is, to route anything at atomist.com through CloudFlare, I have to route atomist.com through CloudFlare. I have to change the routing for my entire company. :-(

Even if I did reroute our whole domain through CloudFlare, the second problem appears: I can get the appearance of security but not actual end-to-end SSL. The easy option to choose is "Flexible", meaning users get SSL from browser<->CloudFlare and it looks secure to them, but behind the scenes it's HTTP between CloudFlare and my app. This seems unprofessional to me, letting everyone's requests happen without SSL behind the scenes while telling them it's secure.

The other option to choose is "Full SSL," but then I need SSL on Cloud Foundry anyway, so ...

SSL in Cloud Foundry 

There's a Pivotal SSL service available in the PWS marketplace for SSL termination. For $20/month (they don't mention that in the documentation), it'll let you upload one certificate.



Currently, we use AWS Certificate Manager, which provides free certificates that only work on AWS.
Can I get that for survey.atomist.com separately, while leaving the rest of atomist.com alone? I'm going to try that, from some other source -- but not today.

Therefore, because our security certificates are tied to AWS
and because I decline to change the routing of our entire domain in order to experiment with this subdomain,
I give up. My toy website doesn't need HTTPS anyway.

The moral is: if you want to experiment with moving part of your infrastructure off of AWS (designated by a subdomain), be prepared to change how requests are routed to the root domain.

Thank you to Glenn Oppegard for information about SSL on PWS, and Simon Brown who is finding this just fine with CloudFlare. 

personal automation example: deleting a local maven artifact

I want to start recording the benefits I get out of personal and local (team-level) automation.

When I have a local version of a maven library installed, say 0.19.0-SNAPSHOT, but I want to go back to using the last released version, say 0.18.0, then I need to delete the library from my local maven repo. I've learned the hard way that it is not enough to delete the directory in the jar; I have to delete another file too. So a while back, I scripted that.

This tiny program deletes a version of the `rug` library, since that's usually the one I'm messing with.

~/bin/delete-version
#!/bin/bash
set -eset -x
version=$1
cd ~/.m2/repositorycd com/atomist/rugls $1rm maven-metadata-local.xmlrm -r $version

This is in ~/bin/delete-version

Today I wanted to remove a version of different library, so I didn't call the program, but I did look at it.

cat `which delete-version`

and then I said oh right. There is the directory to go to and there's that other file I must delete. Even though I haven't automated precisely what I want to do, my automation was useful. 

The only real documentation is code.

Monday, July 10, 2017

See also: Diversity is not a magic pill

After reading Journal of Organizational Psychology (2016 v4), I found an article interesting and summarized it over at the >Code blog:

Diversity is not a magic pill

See also: A Taxonomy of Yak Shaving

After my "Shaving the Golden Yak" talk at Joy of Coding, I published a series of posts about varieties of yaks over at The Composition.

A Taxonomy of Yak Shaving

Thursday, July 6, 2017

Stupid mocha stupid error

For reference only: (and generativity)

This needs to be on the internet, because I didn't find it in a search.

I try to run mocha via npm and pass it `-grep` or `-fgrep` and I get this awful error (can you tell how annoyed I am) about "Cannot find module '-e'" and I think something is wrong with npm install BUT NO

$ npm run mocha -- -fgrep "annotation with string content found"
> @atomist/spring-team-handlers@0.5.10 mocha /Users/jessitron/code/atomisthq/spring-team-handlers/.atomist
> mocha --require intelli-espower-loader 'target/ts-built/mocha/**/*.js' "-fgrep" "annotation with string content found"
Error: Cannot find module '-e'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Module.require (module.js:513:17)
What that should say is

"Invalid option -fgrep. Try --fgrep"

So when future me finds this by searching the internet next time: Jess, use two dashes before the `grep` or `fgrep` options to mocha.

Saturday, June 24, 2017

Hyperproductive development

TL;DR: the most productive development happens when one person knows the system intimately because they wrote it; this is in conflict with growing a system beyond what one person maintains.

Let's talk about why some developers, in some situations, are ten times more productive than others.

hint: it isn't the developers, so much as the situation.

When do we get that exhilarating feeling of hyperproductivity, when new features flow out of our fingertips? It happens when we know our tools like the back of our hands, and more crucially, when we know the systems we are changing. Know them intimately, like I know the contents of my backpack, when I packed it and I tuned the items in each pouch over years of travel. Know the contents of every module, both what they are and what we'd like them to be if we ever finish that refactoring. Know the edges, who uses every API and which changes will break whom, and we're friends with all of the stakeholders. Know the underpinnings, which database fields are indexed and which are obsolete and which have quirky special values. Know the infrastructure, where it runs in production and how to ssh in; where it runs in test and what version is deployed and when it is safe to push a new one. Know the output, what looks normal in the logs and what's a clue. We have scripts, one-liners that tail the logs in all three prod instances to our terminals so our magic eyes can spot the anomaly.

We know it because we wrote it, typically. It is extremely difficult to establish this level of intimacy with an existing system. Braitenberg calls this the Law of Downhill Invention, Uphill Analysis. Complex systems are easier to build than to figure out after they're working.

We know it because we are changing it. The system is alive in our head. It's a kind of symbiosis: we help the system run and grow, and the system works the way we wish. If we walk away for a month or two, begin a relationship with a different system, the magic is lost. It takes time to re-establish familiarity.

Except, I'm suspicious of this description. "We" is a plural pronoun. This depth of familiarity and comfort with a system is personal: it's usually one person. The one person who has conceived this solution, who holds in their head both the current state and where they are aiming.

If you are this person, please realize that no one else experiences this work the way you do. For other people, every change is scary, because they don't know what effect it will have. They spend hours forming theories about how a piece works, and then struggle to confirm this with experiment; they don't have the testing setup you do. They study every log message instead of skimming over the irrelevant ones, the ones you've skipped over so often you don't even see them anymore. By the time they do figure something out, you've changed it; they can't gain comprehension of the system as quickly as you can alter it. When they do make a change, they spend lots of time limiting the scope of it, because they don't know which changes will cause problems. They get it wrong, because they don't know the users personally; communication is hard.

If you are this person, please go easy on everyone else. You are a synthetic biologist and can alter the DNA of the system from within; they are xenosurgeons and have to cut in through the skin and try not to damage unfamiliar organs.

If you work with this person, I'm sorry. This is a tough position to be in, to always feel inferior and like you're breaking everything you touch. I'm there now, in some parts of our system. It's okay for me because the host symbiont, the author and manipulator of that software, is super nice and helpful. He doesn't expect me to work with it the same way he does.

If your team looks like this, here are some steps to take:

  1. Consider: don't change it. This really is the fastest way to develop software. One person, coordinating with no other developers, can move faster than a whole team when the system is small enough. Until! we need the system to grow bigger. Or! the system is crucial to the business (it's an unacceptable risk for only one person to have power over it).
  2. As the host symbiont who lives and breathes the system: strike the words "just", "easy," "obvious," "simple," and "straightforward" from your vocabulary. These words are contextual, and no other human shares your context.
  3. Please write tests. Tests give people who are afraid of unintentional breakages a way to test their theories. Experimentation is crucial to learning how a system works, and tests make experiments possible. They also serve as documentation of intended behavior.
  4. Pair program! By far the best way to transfer understanding of the system to another human is to change it together. (Or boost a whole team at once: mob program!)
  5. Make a README for other developers. Describe the purpose of the system briefly, and document how you develop, test, and troubleshoot the system. Specify the command lines for running tests, for deployment, for accessing logs. Describe in detail how to obtain the necessary passwords. Write down all the environments where it runs, and the protocol around changing them.
  6. Do you know your users better than anyone else? Remedy that. Bring other team members into the discussion. (There's a sweet spot of a single developer-type who works within a business unit. When the software becomes too important for this scale, it gets harder.) Let all the devs get to know the users. Have happy hours. Form redundant communication channels. It'll pay off in ways you never detect.
  7. Slow down. Like seriously, if one person is developing at maximum speed on a project, no one else can get traction. You can't move at full speed and also add symbionts. When it is important to bring in new people, don't do anything alone. Pair on everything. Yes, this will slow you down. It will speed them up. Net, we'll still be slower than you working alone. This is an inherent property of the larger system, which now includes interhuman coordination. There's more overhead than when it was just you and your program. That's OK; it's a tradeoff for safety and scale from sheer speed.
Let's acknowledge that there really are developer+situations that are 10x more productive than others. Let's acknowledge that they don't scale. Make choices about when we can take advantage of the sweet spot of local or individual automation, and when the software we're building is too important for a bus factor of one.
Distinguish between an experimental prototype, when speed of change and redirection is crucial, versus a production app which needs backwards compatibility guarantees and documentation and all that seriousness -- this requires a solid team.

Recognize that the most productive circumstance for development is a rare circumstance. Most of the time, I need to work on a system that someone else wrote. (that "someone else" could be "me, months or years ago.") The temptation to rewrite is strong, because if I rewrite it then I'll understand it.

There's a time for 10x development, and a time for team development. When you want to be serious, the 10x developer prevents this. If that's your situation, please consider the suggestions in this post.