Unit tests and assertions (was: [Exceptional C++ Style] Item 18 - Virtuality)

[Kevlin Henney]

In message <018a01c4f64b$1e6fccc0$0300000a at pc>, Terje Slettebø 
<tslettebo at broadpark.no> writes
>>From: "Kevlin Henney" <kevlin at curbralan.com>
>
>> In addition, introducing such a DbC framework is generally a waste of
>> effort. Although it is based on good intentions, it is mostly ritual
>> rather than time well invested. A set of unit tests will be a much
>> clearer guide to the contract and the interface. Wading through an NVI
>> version of a contract is more than a little tedious, especially when the
>> notion of contract enforcement is based on assertions: all that
>> infrastructure for something that isn't even compiled in.
>
>Unless I misunderstand you, aren't unit tests and asserts rather
>complementary to each other? The asserts checks
>preconditions/postconditions/invariants, while you may use unit tests to
>check that they indeed work, i.e. exercise the asserts.

When a contract is phrased so that it does not describe what happens 
when the normal terms of a contract are violated, the assumed behaviour 
is undefined, ie the program may terminate, an exception may be thrown, 
the called function need never terminate, everything may continue as 
normal, etc. A contract phrased like this could be checked (or ignored) 
quite legitimately through 'assert' (as in the <cassert> macro). 
However, there is nothing for a test case to assert in such a case.

Where a contract also publishes the specific quality of failure for 
transgression of some or all of its normal parts, because such 
behaviours are published in terms of definite measurable outcomes, they 
may be tested for.

What example-based unit test cases allow you to do is present and check 
the items specified in the contract associated with a measurable 
outcome. In other words, specific normal outcomes in valid situations 
and specified bad-news in unpleasant circumstances.

This covers one of the most significant blindspots activities in the use 
of assertions in core code: asserting on the correctness of the return 
values or outcome, ie the postcondition. Writing such assertions is 
often too trivial to demonstrate anything, hence they are omitted 
(consider the postcondition for a container's empty function: for 
queries the postcondition can often be the definition), or they are 
often non-trivial to write, hence they are omitted (consider the 
postcondition of a sorting function). Example-based test cases are 
specific and illustrative, and can deal with accumulation of state 
changes -- predicate calculus has no concept of history.

When I was first introduced to design by contract <mumble> years ago, I 
was taken by the whole concept, but the principle/principal weakness 
appeared to be in writing assertions for postconditions. The notion of a 
contract is one of the most powerful interface design tools available, 
but we must be careful to separate that idea from what that means for 
code -- DbC does not entail asserts. Postconditions define what an 
outcome must satisfy, but in their generalised form they are not good 
tests of an implementation.

Preconditions fall into two broad groups:

(1) Preconditions with no associated penalty terms for violation, ie 
undefined behaviour. These cannot be tested for in unit test cases. They 
test the correctness of the caller rather than the called code, so there 
is nothing to test at this level.

(2) Preconditions associated with some form of well-defined penalty, eg 
an exception. With the exception of a contract that leads to process 
death, these can be tested for using unit tests.

In case (1) preconditions that lead to undefined behaviour can be 
further subdivided into three groups:

(i) Preconditions for which it is not possible to write an assertion 
within the production code, eg "the passed iterator must be valid".

(ii) Preconditions for which introducing an assertion is intrusive, 
either in terms of the effort involved or in terms of its runtime 
effect, eg "the passed data set must be normalised".

(iii) Preconditions for which assertions are trivial to write.

Naturally enough, people tend to target (iii), and there are certainly 
many cases in this class. However, getting carried away with (iii) tends 
to block recognition of (2), cases where it is the component's 
responsibility to do -- and publish -- something more appropriate than 
just the possibility of falling over.

Where case (iii) can become useful as written assertions is in the 
context of integration testing. In such a situation, above the level of 
the unit, then testing the caller under certain circumstances does make 
sense and arises as a natural consequence of integration. However, this 
does not automatically mean that introducing 'assert' in such cases is 
appropriate.

Perhaps the form of assertion that comes off best is in support of 
data-structure (eg class invariant) or control-flow invariants (eg loop 
invariant). Often these are truly matters of internal structure that are 
not appropriate for the outside world to see (eg this internal pointer 
is never null) and is white-box knowledge that should not leak into 
black-box tests. They also follow (i), (ii) and (iii) in terms of 
applicability, but are things that can in principle be revealed by unit 
tests and not just integration tests.

And then, of course, there are all the contracts that cannot be stated 
fully and sensibly in terms of pre- and postconditions. For example, the 
contract that binds the behaviour of Object.{e,E}quals in {Java, .NET}, 
or contracts in the context of re-entrant code, such as callbacks.

So, yes, unit testing is often complementary to assertions, but 
sometimes they impinge on the same concerns. Programmers may consider 
the wrong path for their specific concerns. Many who plaster 'assert' 
over their code do so motivated by concerns that are better addressed 
with unit tests and -- but to a (much) lesser degree -- vice versa.

Kevlin
-- 
____________________________________________________________

   Kevlin Henney                   phone:  +44 117 942 2990
   mailto:kevlin at curbralan.com     mobile: +44 7801 073 508
   http://www.curbralan.com        fax:    +44 870 052 2289
   Curbralan: Consultancy + Training + Development + Review
____________________________________________________________