Ultratestable Coding Style

Darn side-effecting programs. Programs that change things in the outside world are so darn useful, and such a pain to test.
what's better than green? Ultra!For every piece of code, there is another piece of code that answers the question, “How do I know that code works?” Sometimes that’s more work than the code itself — but there is hope.

The other day, I made a program to copy some code from one project to another – two file copies, with one small change to the namespace declaration at the top of each file. Sounds trivial, right?

I know better: there are going to be a lot of subtleties. And this isn’t throwaway code. I need good, repeatable tests.

Where do I start? Hmm, I’ll need a destination directory with the expected structure, an empty source directory, files with the namespace at the top… oh, and cleanup code. All of these are harder than I expected, and the one test I did manage to write is specific to my filesystem. Writing code to verify code is so much harder than just writing the code!

Testing side-effecting code is hard. This is well established. It’s also convoluted, complex, generally brittle.
The test process looks like this:

input to code under test to output, but also prep the files in the right place and clear old files out, then the code under test does read & write on the filesystem, then check that the files are correct

Before the test, create the input AND go to the filesystem, prepare the input and the spot where output is expected.
After the test, check the output AND go to the filesystem, read the files from there and check their contents.
Everything is intertwined: the prep, the implementation of the code under test, and the checks at the end. It’s specific to my filesystem. And it’s slow. No way can I run more than a few of these each build.

The usual solution to this is to mock the filesystem. Use a ports-and-adapters approach. In OO you might use dependency injection; in FP you’d pass functions in for “how to read” and “how to write.” This isolates our code from the real filesystem. Test are faster and less tightly coupled to the environment. The test process looks like this:

Before the test, create the input AND prepare the mock read results and initialize the mock for write captures.
After the test, check the output AND interrogate the mock for write captures.

It’s an improvement, but we can do better. The test is still convoluted. Elaborate mocking frameworks might make it cleaner, but conceptually, all those ties are still there, with the stateful how-to-write that we pass in and then ask later, “What were your experiences during this test?”

If I move the side effects out of the code under test — gather all input beforehand, perform all writes afterward — then the decisionmaking part of my program becomes easier and more clear to test. It can look like this (code):

The input includes everything my decisions need to know from the filesystem: the destination directory and list of all files in it; the source directory and list plus contents of all files in it.
The output includes a list of instructions, for the side effects the code would like to perform. This is super easy to check at the end of a test.

The real main method looks different in this design. It has to gather all the input up front[1], then call the key program logic, then carry out the instructions. In order to keep all the decisionmaking, parsing, etc in the “code under test” block, I keep the interface to that function as close as possible to that of the built-in filesystem-interaction commands. It isn’t the cleanest interface, but I want all the parts outside “code-under-test” to be trivial.

simplest possible code to gather input, to well-tested code that makes all the decisions, to simplest-possible code to carry out instructions.

With this, I answer “How do I know this code works?” in two components. For the real-filesystem interactions, the documentation plus some playing around in the REPL tell me how they work. For the decisioning part of the program, my tests tell me it works. Manual tests for the hard-to-test bits, lots of tests for the hard-to-get-right bits. Reasoning glues them together.

Of course, I’m keeping my one umbrella test that interacts with the real filesystem. The decisioning part of the program is covered by poncho tests. With an interface like this, I can write property-based tests for my program, asserting things like “I never try to write a file in a directory that doesn’t exist” and “the output filename always matches the input filename.”[2]

As a major bonus, error handling becomes more modular. If, on trying to copy the second file, it isn’t found or isn’t valid, the second write instruction is replaced with an “error” instruction. Before any instructions are carried out, the program checks for “error” anywhere in the list (code). If found, stop before carrying out any real action. This way, validations aren’t separated in code from the operations they apply to, and yet all validations happen before operations are carried out. Real stuff happens only when all instructions are possible (as far as the program can tell). It’s close to atomic.

There are limitations to this straightforward approach to isolating decisions from side-effects. It works for this program because it can gather all the input, produce all the output, and hold all of it in memory at the same time. For a more general approach to this same goal, see Functional Programming in Scala.

Moving all the “what does the world around me look like?” side effects to the beginning of the program, and all the “change the world around me!” side effects to the end of the program, we achieve maximum testability of program logic. And minimum convolution. And separation of concerns: one module makes the decisions, another one carries them out. Consider this possibility the next time you find yourself in testing pain.


The code that inspired this approach is in my microlib repository.
Interesting bits:
Umbrella test (integration)
Poncho tests (around the decisioning module) (I only wrote a few. It’s still a play project right now.)
Code under test (decisioning module)
Main program
Instruction carrying-out part

Diagrams made with Monodraw. Wanted to paste them in as ASCII instead of screenshots, but that’d be crap on mobile.


[1] This is Clojure, so I put the “contents of each file” in a delay. Files whose contents are not needed are never opened.
[2] I haven’t written property tests, because time.

Fun with Optional Typing: cheap mocking

For unit tests, it’s handy to mock out side-effecting functions so they don’t slow down tests.[1] Clojure has an easy way to do this: use with-redefs to override function definitions, and then any code within the with-redefs block uses those definitions instead.

To verify the input of the side-effecting function, I can override it with something that throws an exception if the input is wrong.[2] A quick way to do that is to check the input against a schema.

That turns out to be kinda pretty. For instance, if I need to override this function fetch-orders, I can enforce that it receives exactly the starting-date I expect, and a second argument that is not specified precisely, but still meets a certain condition.

(with-redefs [fetch-orders (s/fn [s :- (s/eq starting-date)
                                  e :- AtLeastAnHourAgo]
                            [order])]
… )

Here, the s/fn macro creates a function that (when validation is activated[3]) checks its input against the schemas specified after the bird-face operator. The “equals” schema-creating function is built-in; the other I created myself with a descriptive name. The overriding function is declarative, no conditionals or explicit throwing or saving mutable state for later.

If I have a bug that switches the order of the inputs, this test fails. The exception that comes out isn’t pretty.

expected: (= expected-result (apply function-under-test input))
  actual: clojure.lang.ExceptionInfo: Input to fn3181 does not match schema: [(named (not (= # a-org.joda.time.DateTime)) s) nil]

Schema isn’t there yet on pretty errors. But hey, my test reads cleanly, it was simple to write, and I didn’t bring in a mocking framework.

See the full code (in the literate-test sort of style I’m experimenting with) on github.


[1] for the record, I much prefer writing code that’s a pipeline, so that I only have to unit-test data-in, data-out functions. Then side-effecting functions are only tested in integration tests, not mocked at all. But this was someone else’s code I was adding tests around.

[2] Another way to check the output is to have the override put its input into an atom, then check what happened during the assertion portion of the test. Sometimes that is cleaner.

[3] Don’t forget to (use-fixtures :once schema.test/validate-schemas) 

Fun with Optional Typing: narrowing errors

After moving from Scala to Clojure, I miss the types. Lately I’ve been playing with Prismatic Schema, a sort of optional typing mechanism for Clojure. It has some surprising benefits, even over Scala’s typing sometimes. I plan some posts about interesting ones of those, but first a more ordinary use of types: locating errors.

Today I got an error in a test, and struggled to figure it out. It looked like this:[1]

expected: (= [expected-conversion] result)
  actual: (not (= [{:click {:who {:uuid “aeiou”}, :when #}, :outcome {:who {:uuid “aeiou”}, :when #, :what “bought 3 things”}}] ([{:click {:who {:uuid “aeiou”}, :when #}, :outcome {:who {:uuid “aeiou”}, :when #, :what “bought 3 things”}}])))

Hideous, right? It’s super hard to see what’s different between the expected and actual there. (The colors help, but the terminal doesn’t give me those.)

It’s hard to find the difference because the difference isn’t content: it’s type. I expected a vector of a map, and got a list of a vector of a map. Joy.

I went back and added a few schemas to my functions, and the error changed to

  actual: clojure.lang.ExceptionInfo: Output of calculate-conversions-since does not match schema: [(not (map? a-clojure.lang.PersistentVector))]

This says my function output was a vector of a vector instead of a map. (This is one of Schema’s more readable error messages.)

Turns out (concat (something that returns a vector)) doesn’t do much; I needed to (apply concat to-the-vector).[2]

Clojure lets me keep the types in my head for as long as I want. Schema lets me write them down when they start to get out of hand, and uses them to narrow down where an error is. Even after I spotted the extra layer of sequence in my output, it could have been in a few places. Adding schemas pointed me directly to the function that wasn’t doing what I expected.

The real point of types is that they clarify my thinking and document it at the same time. They are a skeleton for my program. I like Clojure+Schema because it lets me start with a flexible pile of clay, and add bones as they’re needed.

—–
[1] It would be less ugly if humane-test-output were activated, but I’m having technical difficulties with that at the moment.
[2] here’s the commit with the schemas and the fix.

My First Leiningen Template

Every time I sit down to write a quick piece of code for a blog post, it starts with “lein new.” This is amazing and wonderful: it’s super fast to set up a clean project. Good practice, good play.[1]

But not fast enough! I usually start with a property-based test, so the first thing I do every time is add test.check to the classpath, and import generators and properties and defspec in the test file. And now that I’ve got the hang of declaring input and output types with prismatic.schema, I want that everywhere too.

I can’t bring myself to do this again – it’s time to shave the yak and make my own leiningen template.

The instructions are good, but there are some quirks. Here’s how to make your own personal template, bringing your own favorite libraries in every time.

It’s less confusing if the template project directory is not exactly the template name, so start with:

  lein new template your-name –to-dir your-name-template
  cd your-name-template

Next, all the files in that directory are boring. Pretty them up if you want, but the meat is down in src/leiningen/new.

In src/leiningen/new/your-name.clj is the code that will create the project when your template is activated. This is where you’ll calculate anything you need to include in your template, and render files into the right location. The template template gives you one that’s pretty useless, so I dug into leiningen’s code to steal and modify the default template’s definition. Here’s mine:

(defn jessitron
 [name]
 (let [data {:name name
             :sanitized (sanitize name)
             :year (year)}]
  (main/info “Generating fresh project with test.check and schema.“)
  (->files data
     [“src/{{sanitized}}/core.clj” (render “core.clj” data)]
     [“project.clj” (render “project.clj” data)]
     [“README.md” (render “README.md” data)]
     [“LICENSE” (render “LICENSE” data)]
     [“.gitignore” (render “gitignore” data)]
     [“test/{{sanitized}}/core_test.clj” (render “test.clj” data)]))

As input, we get the name of the project that someone is creating with our template.
The data map contains information available to the templates: that’s both the destination file names and the initial file contents. Put whatever you like in here.
Then, set the message that will appear when you use the template.
Finally, there’s a vector of destinations, paired with renderings from source templates.

Next, find the template files in src/leiningen/new/your-name/. By default, there’s only one. I stole the ones leiningen uses for the default template, from here. They didn’t work for me immediately, though: they referenced some data, such as {{namespace}}, that wasn’t in the data map. Dunno how that works in real life; I changed them to use {{name}} and other items provided in the data.

When it’s time to test, two choices: go to the root of your template directory, and use it.

lein new your-name shiny-new-project

This feels weird, calling lein new within a project, but it works. Now
cd shiny-new-project
lein test

and check for problems. Delete, change the template, try again.

Once it works, you’ll want to use the template outside the template project. To get this to work, first edit project.clj, and remove -SNAPSHOT from the project version.[3] Then

lein install

Done! From now on I can lein new your-name shiny-new-project all day long.

And now that I have it, maybe I’ll get back to the post I was trying to write when I refused to add test.check manually one last time.


[1] Please please will somebody make this for sbt? Starting a Scala project is a pain in the arse[2] compared to “lein new,” which leans me toward Clojure over Scala for toy projects, and therefore real projects.

[2] and don’t say use IntelliJ, it’s even more painful there to start a new Scala project.

[3] At least for me, this was necessary. lein install didn’t get it into my classpath until I declared it a real (non-snapshot) version.

Logs are like onions

Or, What underlying implementation is clojure.tools.logging using?

Today I want to change the logging configuration of a Clojure program. Where is that configuration located? Changing the obvious resources/log4j.properties doesn’t seem to change the program’s behavior.

The program uses clojure.tools.logging, but that’s a wrapper around four different underlying implementations. Each of those implementations has its own ideas about configuration. How can I find out which one it uses?

Add a println to your program[1] to output this:

(.name clojure.tools.logging/*logger-factory*)         

In my case the output is:

org.slf4j

This is clojure logging’s first choice of factories. If it can instantiate this, it’ll use it. Now I can google slf4j and find that it… is also a facade on top of multiple logging implementations.
Digging into the slf4j source code reveals this trick:

(class (org.slf4j.LoggerFactory/getILoggerFactory)) 

which prints:

org.slf4j.impl.Log4jLoggerFactory

so hey! I am using log4j after all! Now why doesn’t it pick up resources/log4j.properties?
Crawling through the log4j 1.2 (slf4j seems to use this version) source code suggests this[2]:

(org.apache.log4j.helpers.Loader/getResource “log4j.properties”)

which gives me

#

So hey, I finally have a way to trace where logging configuration comes from! 
In the end, my guess of resources/log4j.properties was correct. I forgot to rebuild the uberjar that I was running. The uberjar found the properties file in itself:

jar:file:/Users/jessitron/…/target/program-0.1.0-SNAPSHOT-standalone.jar!/log4j.properties

Bet I’d have realized that a few hours earlier if I were pairing today. And then I wouldn’t have made this lovely post.

———-
[1] or run it in the cider REPL in emacs, in your namespace
[2] actually it checks for log4j.xml first; if that’s found it’ll choose the xml file over the .properties.

A victory for abstraction, re-use, and small libraries

The other day at Outpace, while breaking some coupling, Eli and I decided to retain some information from one run of our program to another. We need to bookmark how far we read in each input data table. How can we persist this small piece of data?

Let’s put it in a file. Sure, that’ll work.[1] 

Next step, make an abstraction. Each of three configurations needs its own “how to read the bookmark” and “how to write the bookmark.”[2] What can we name it?

After some discussion we notice this is basically a Clojure atom – a “place” to store data that can change – except persistent between runs.

Eli googles “clojure persist atom to disk” and bam! He finds a library. Enduro, by @alandipert. Persistent atoms for Clojure, backed by a file or Postgres. Complete with an in-memory implementation for testing. And thread safety, which we would not have bothered with. Hey, come to think of it, Postgres is a better place to store our bookmarks.

From a need to an abstraction to an existing implementation! with better ideas! win!

Enduro has no commits in the last year, but who cares? When a library is small enough, it reaches feature-completion. For a solid abstraction, there is such a thing as “done.”

Now, it happens that the library isn’t as complete as we hoped. There are no tests for the Postgres implementation. The release! method mentioned in the README doesn’t exist.

But hey, we can add these to the library faster and with less risk than implementing it all ourselves. Alan’s design is better than ours. Building on a solid foundation from an expert is more satisfying that building from scratch. And with pull requests, everybody wins!

This is re-use at its best. We paused to concentrate on abstraction before implementation, and it paid off.


[1] If something happens to the file, our program will require a command-line argument to tell it where to start.

[2] In OO, I’d put that in an object, implementing two single-method interfaces for ISP, since each function is needed in a different part of the program. In Clojure, I’m more inclined to create a pair of functions. Without types, though, it’s hard to see the meaning of the two disparate elements of the pair. The best we come up with is JavaScript-object-style: a map containing :read-fn and :write-fn. At least that gives them names.

Property tests don’t have to be generative

Now and then, a property test can be easier than an example test. Today, Tanya and I benefited.

There’s this web service. It returns a whole tree of information, some of it useful and some of it is not.

{ “category”: “food”,
  “children: [ { “category” : “fruit”,
                  “children” : […LOTS MORE…],
                  “updatedAt” : “2014-06-30T16:22:36.440Z”,
                  “createdAt” : “2014-06-30T16:22:36.440Z”},
                 {“category” : “vegetables”,
                  “children” : […EVEN MORE…],
                  “updatedAt” : “2014-06-25T18:32:36.436Z”,
                  “createdAt” : “2014-06-25T18:32:36.436Z”}],
  “updatedAt” : “2014-06-15T16:32:36.550Z”,
  “createdAt” : “2014-03-05T08:12:46.440Z” }

The service is taking a while, mostly because it’s returning half a meg of data. Removing the useless fields will cut that in half.

Being good developers, we want to start with a failing test. An example test might start with inserting data in the database, perhaps after clearing the table out, so we can know what the service should return. That’s a lot of work, when I don’t really care what’s returned, as long as it doesn’t include those updatedAt and createdAt fields.

Currently when we test the function implementing this service, there’s some sample data lying around. If we write a property test instead of an example test, that data is good enough. As long as the service returns some data, and it’s something with children so we can check nested values, it’s good enough. I can test that: (actual code)

(deftest streamline-returned-tree
  (testing “boring fields are not returned”
    (let [result (method-under-test)]
      (is (seq (:children result))))))

This is not a generative test, because it doesn’t generate its own data, and it doesn’t run repeatedly. Yet it is a property test, because it’s asserting a property of the result. The test doesn’t say “expected equals actual.” Instead, it says “The result has children.”

This test passes, and now we can add the property of interest: that the map returned by method-under-test has no :createdAt or :updatedAt keys, at any level. We could find or write a recursive function to dig around in the maps and nested vectors of maps, but that same function would also be useful in the implementation. Duplicating that code in the test is no good.

One of the classic challenges of property testing is finding two different ways to do the same thing. Then we can make assertions without knowing the input. But… nobody said they have to be two smart ways of doing the same thing! I want to be sure there’s no “createdAt blah blah” in the output, so how about we write that nested map to a string and look for “createdAt” in that?

(deftest streamline-returned-tree
  (testing “boring fields are not returned”
    (let [result (method-under-test)
          result-string (pr-str result)]
      (is (seq (:children result)))
      (is (not (.contains result-string:createdAt“))))))

This gives us a failing test, and it was a heck of a lot easier to implement than an example test which hard-codes expected results. This test is specific about its purpose. As a bonus, it doesn’t use any strategy we’d ever consider using in the implementation. The print-it-to-a-string idea, which sounded stupid at first, expresses the intention of “we don’t want this stuff included.”

Property tests don’t have to be generative, and they don’t have to be clever. Sometimes it’s the dumb tests that work the best.


Bonus material:
the output of this failing test is

expected: (not (.contains result-string:createdAt“))
  actual: (not (not true))

This “not not true” actual result… yeah, not super useful. clojure-test’s “is” macro responds better to a comparison function than to nested calls. If I define a function not-contains, then I can get:

expected: (not-contains result-string “:createdAt“)
  actual: (not (not-contains 
                “{:children [{:createdAt \”yesterday\”, :category \”fruit\”}], :createdAt \”last week\”, :category \”food\”}” 
                “:createdAt“))

That’s a little more useful, since it shows what it’s comparing.

A monadically built generator

Today, I wanted to write a post about code that sorts a vector of maps. But I can’t write that without a test, now can I? And not just any test — a property-based test! I want to be sure my function works all the time, for all valid input. Also, I don’t want to come up with representative examples – that’s too much work.[1]

The function under test is a custom-sort function, which accepts a bunch of rows (represented as a sequence of hashmaps) and a sequence of instructions: “sort by the value of A, descending; then the value of B, ascending.”

To test with all valid input, I must write code to generate all valid input. I need a vector of maps. The maps should have all the same keys. Some of those keys will be sort instructions. The values in the map can be anything Comparable: strings and ints for instance. Each instructions also includes a direction, ascending or descending. That’s a lot to put together.

For property-based (or “generative”) tests in Clojure, I’ll use test.check. To test a property, I must write a generator that produces input. How do I even start to create a generator this complicated?

Bit by bit! Start with the keys for the maps. Test.check has a generator for them:

(require ‘[clojure.test.check.generators :as gen])
gen/keyword ;; any valid clojure keyword.

The zeroth secret: I dug around in the source to find useful generators. If it seems like I’m pulling these out of my butt, well, this is what I ate.

Next I need multiple keywords, so add in gen/vector. It’s a function that takes a generator as an argument, and uses that repeatedly to create each element, producing a vector.

(gen/vector gen/keyword) ;; between 0 and some keywords

The first secret: generator composition. Put two together, get a better one out.

Since I want a set of keys, not a vector, it’s time for gen/fmap (“functor map,” as opposed to hashmap). That takes a function to run on each produced value before giving it to me, and its source generator.

(gen/fmap set (gen/vector gen/keyword)) ;; set of 0 or more keywords

It wouldn’t do for that set to be empty; my function requires at least 1 instruction, which means at least one keyword. gen/such-that narrows the possible output of the generator. It takes a predicate and a source generator:

(gen/such-that seq (gen/fmap set (gen/vector gen/keyword)))

If you’re not a seasoned Clojure dev: seq is idiomatic for “not empty.” Historical reasons.

This is enough to give me a set of keys, but it’s confusing, so I’m going to pull some of it out into a named function.

(defn non-empty-set [elem-g
  (gen/such-that seq (gen/fmap set (gen/vector elem-g))))

Here’s the generator so far:
(def maps-and-sort-instructions
  (let [set-of-keys  (non-empty-set gen/keyword)]
     set-of-keys)

See what it gives me:
=> (gen/sample maps-and-sort-instructions
   ;; sample makes the generator produce ten values
(#{:Os} #{:? :f_Q_:_kpY:+:518} #{:? :-kZ:9_:_?Ok:JS?F} ….)

Ew. Nasty keywords I never would have come up with. But hey, they’re sets and they’re not empty.

To get maps, I need gen/hash-map. It wants keys, plus generators that produce values; from these it produces maps with a consistent structure, just like I want. It looks like:

(gen/hash-map :one-key gen-of-value :two-key gen-of-this-other-value …)

The value for each key could be anything Comparable really; I’ll settle for strings or ints. Later I can add more to this list. There’s gen/string and gen/int for those; I can choose among them with gen/elements.

(gen/elements [gen/string gen/int]) ;; one of the values in the input vector

I have now created a generator of generators. gen/elements is good for selecting randomly among a known sequence of values. I need a quantity of these value generators, the same quantity as I have keys.

(gen/vector (gen/elements [gen/string gen/int]) (count #??#)) 
  ;; gen/vector accepts an optional length

Well, crap. Now I have a dependency on what I already generated. Test.check alone doesn’t make this easy – you can do it, with some ugly use of gen/bind. Monads to the rescue! With a little plumbing, I can bring in algo.monad, and make the value produced from each generator available to the ones declared after it.

The second secret: monads let generators depend on each others’ output.

(require ‘[clojure.algo.monads :as m])
(m/defmonad gen-m
    [m-bind gen/bind
     m-result gen/return])

(def maps-and-sort-instructions
 (m/domonad gen-m
   [set-of-keys (non-empty-set gen/keyword)
    set-of-value-gens (gen/vector  
                       (gen/elements [gen/string gen/int]) 
                       (count set-of-keys))]
    [set-of-keys, set-of-value-gens])

I don’t recommend sampling this; generators don’t have nice toStrings. It’s time to put those keys and value-generators together, and pass them to gen/hash-map:

(apply gen/hash-map (mapcat vector set-of-keys set-of-value-generators))
  ;; intersperse keys and value-gens, then pass them to gen/hash-map

That’s a generator of maps. We need 0 or more maps, so here comes gen/vector again:

(def maps-and-sort-instructions
 (m/domonad gen-m
  [set-of-keys (non-empty-set gen/keyword)
   set-of-value-gens (gen/vector  
                      (gen/elements [gen/string gen/int]) 
                      (count set-of-keys))
   some-maps (gen/vector 
              (apply gen/hash-map 
               (mapcat vector set-of-keys 
                              set-of-value-gens)))]
  some-maps))

This is worth sampling a few times:
=> (gen/sample maps-and-sort-instructions 3) ;; produce 3 values
([] [] [{:!6!:t4 “à$”, :*B 2, :K0:R*Hw:g:4!? “”}])

It randomly produced two empty vectors first, which is fine. It’s valid to sort 0 maps. If I run that sample more, I’ll see vectors with more maps in them.
Halfway there! Now for the instructions. Start with a subset of the map keys – there’s no subset generator, but I can build one using the non-empty-set defined earlier. I want a non-empty-set of elements from my set-of-keys.

(non-empty-set (gen/elements set-of-keys)) 
  ;; some-keys: 1 or more keys. 

To pair these instruction keys with directions, I’ll generate the right number of directions. Generating a direction means choosing between :ascending or :descending. This is a smaller generator that I can define outside:

(def mygen-direction-of-sort 
      (gen/elements [:ascending :descending])) 

and then to get a specific-length vector of these:

(gen/vector mygen-direction-of-sort (count some-keys)) 
   ;; some-directions

I’ll put the instruction keys with the directions together after the generation is all complete, and assemble the output:

(def maps-and-sort-instructions
 (m/domonad gen-m
  [set-of-keys (non-empty-set gen/keyword)
   set-of-value-gens (gen/vector  
                      (gen/elements [gen/string gen/int]) 
                      (count set-of-keys))
   some-maps (gen/vector 
              (apply gen/hash-map 
               (mapcat vector set-of-keys 
                              set-of-value-gens)))
   some-keys (non-empty-set (gen/elements set-of-keys)) 
   some-directions (gen/vector mygen-direction-of-sort 
                               (count some-keys))]
        
   (let [instructions (map vector some-keys some-directions)] 
                           ;; pair keys with directions
    [some-maps instructions]))) ;; return maps and instructions

There it is, one giant generator, built of at least 11 small ones. That’s a lot of Clojure code… way too much to trust without a test. I need a property for my generator!
What is important about the output of this generator? Every instruction is a pair, every direction is either :ascending or :descending, and every key in the sort instructions is present in every map. I could also specify that the values for each key are all Comparable with each other, but I haven’t yet. This is close enough:

(def sort-instructions-are-compatible-with-maps
  (prop/for-all
    [[rows instructions] maps-and-sort-instructions]
    (every? identity (for [[k direction] instructions
                          ;; break instructions into parts
              (and (#{:ascending :descending} direction
                    ;; Clojure looks so weird
                   (every? k rows)))))) 
                          ;; will be false if the key is absent

(require ‘[clojure.test.check :as tc])
(tc/quick-check 50 sort-instructions-are-compatible-with-maps)
;; {:result true, :num-tests 50, :seed 1412659276160}

Hurray, my property is true. My generator works. Now I can write a test… then maybe the code… then someday the post that I wanted to write tonight.

You might roll your eyes at me for going to these lengths to test code that’s only going to be used in a blog post. But I want code that works, not just two or three times but all the time. (Write enough concurrent code, and you notice the a difference between “I saw it work” and “it works.”) Since I’m working in Clojure, I can’t lean on the compiler to test the skeleton of my program. It’s all on me. And “I saw it work once in the REPL” isn’t satisfying.

Blake Meike points out on Twitter, “Nearly the entire Internet revolution… is based on works-so-far code.” So true! It’s that way at work. Maybe my free-time coding is the only coding I get to do right. Maybe that’s why open-source software has the potential to be more correct than commercial software. Maybe it’s the late-night principles of a few hungry-for-correctness programmers that move technology forward.

Nah.

But it does feel good to write a solid property-based test.

————-
[1] Coming up with examples is “work,” as opposed to “programming.”

Code for this post: https://github.com/jessitron/sortificate/blob/generator-post/test/sortificate/core_test.clj

A suggestion for testing style in Clojure

Getting a test to fail is one thing; getting it to express why it failed is another.
Clojure.test provides an assertion macro: is

(deftest “my-function” 
  (testing “some requirement”
    (is (something-that-evaluates-to-bool (arg1) (arg2)) 
    “Message to print when it’s false”)))

When this assertion fails, in addition to the message, “expected” and “actual” results print. The is macro tries to be a little clever.

If the expression passed to is an S-expr, and the first element of the is recognized as a function. Then is prints that first symbol directly, then evaluates all the arguments to the function and prints the results. For instance:

expected: (function-name (arg1) (arg2))
actual: (not (function-name “1st arg value” “2nd arg value”))

However, if is does not recognize that first element as a function, the whole expression passed to is is evaluated for the actual, and you get:

expected: (something-that-evaluates-to-bool (arg1) (arg2))
actual: false

To get the more communicative output, write a function with a name that describes the property you’re testing. A top-level defn will be recognized by is as a function, so declare your test function that way.

For instance, if I’m testing whether a vector of numbers is sorted, I could use built-in functions:

(let [v [1 3 2 4]]
(is (apply <= v) “not sorted”))

Then I see[1]:

expected: (apply <= v)
actual: (not
           (apply
            #
            [1 3 2 4]))

Printing a function is never pretty. If, instead, I give the desired property a name:

(defn ascending? [v] (apply <= v))

(let [v [1 3 2 4]]
  (is (ascending? v) “not sorted”))

Then I get something much more communicative:

expected: (ascending? v)
actual: (not (ascending? [1 3 2 4]))

There it is — declarative test output through creation of simple functions.

Bonus: If you want to make your failure output even better, define special functionality just for your function, as @pjstadig did for =.

—————–
[1] I’m using humane-test-output to get the “actual” pretty-printed.

Clojure