Tests and Assertions

The other day, Will claimed that you should have one assertion for every test case in a test suite. Ryan protested, what if you have an imperative sequence that you want to check for various properties at different stages of the computation? Shouldn't this be conceptually a single test with multiple assertions? According to Will, this should be several tests, each with one assertion; the parts of the script that are common to each of the tests should of course be abstracted (with macros where necessary), but each of these properties is a separate test.

We all agree that this should be abstracted. But the question is, where does the abstraction belong? Should it be custom-made in each case by the user, or should there be a library?

It seems like a common pattern that you'd like to be able to say

(define-script
  (do-something-1 ...)
  (assert property-1 ...)
  (do-something-2 ...)
  (assert property-2 ...)
  (do-something-3 ...)
  (assert property-3 ...)
  ...)

Now, according to Will, this should be generating multiple test cases, with one test for each assertion. In that case, the above pseudocode would ostensibly produce a test-suite value, with multiple test-case values embedded within it.

Another way to look at it, which Ryan suggested, is that you could have multiple assertions per test case. This brings up other questions: what happens if you never reach an assertion because an earlier part of the test raised an exception? Do you count it as a failure? (If so, do you need a static analysis that can predict all the places where assertions might be made?) Do you just ignore it and say it doesn't exist?

Either way, I still think this is an important enough abstraction to belong in the SchemeUnit library. I'm just not quite sure how it should look.