[Exceptional C++ Style] Re: Unit tests and assertions

[James Dennett]

On Monday, January 10, 2005, at 09:54 AM, Jon Jagger wrote:

> Kevlin Henney wrote:
>
>> In message <018a01c4f64b$1e6fccc0$0300000a at pc>, Terje Slettebø 
>> <tslettebo at broadpark.no> writes
>>
>>>> From: "Kevlin Henney" <kevlin at curbralan.com>
>>>
>>>
>>>> In addition, introducing such a DbC framework is generally a waste 
>>>> of
>>>> effort. Although it is based on good intentions, it is mostly ritual
>>>> rather than time well invested. A set of unit tests will be a much
>>>> clearer guide to the contract and the interface. Wading through an 
>>>> NVI
>>>> version of a contract is more than a little tedious, especially 
>>>> when the
>>>> notion of contract enforcement is based on assertions: all that
>>>> infrastructure for something that isn't even compiled in.
>>>
>>>
>>> Unless I misunderstand you, aren't unit tests and asserts rather
>>> complementary to each other? The asserts checks
>>> preconditions/postconditions/invariants, while you may use unit 
>>> tests to
>>> check that they indeed work, i.e. exercise the asserts.
>>
>>
>> When a contract is phrased so that it does not describe what happens 
>> when the normal terms of a contract are violated, the assumed 
>> behaviour is undefined, ie the program may terminate, an exception 
>> may be thrown, the called function need never terminate, everything 
>> may continue as normal, etc. A contract phrased like this could be 
>> checked (or ignored) quite legitimately through 'assert' (as in the 
>> <cassert> macro). However, there is nothing for a test case to assert 
>> in such a case.
>>
>> Where a contract also publishes the specific quality of failure for 
>> transgression of some or all of its normal parts, because such 
>> behaviours are published in terms of definite measurable outcomes, 
>> they may be tested for.
>>
>> What example-based unit test cases allow you to do is present and 
>> check the items specified in the contract associated with a 
>> measurable outcome. In other words, specific normal outcomes in valid 
>> situations and specified bad-news in unpleasant circumstances.
>>
> So, turning the discussion on its head a little, a natural question is 
> when should a contract specify only the behaviour when the 
> preconditions are met, and when should it also specify (to some 
[ *** too many quoted lines.  automatically truncated *** ]


Not as trivial as it seems.  Concrete situations yield fairly clear 
answers,
but I can't easily give blanket rules.

> Failed preconditions are not "functional" but they can be 
> "operational". For example, it would be a bad move to not test 
> preconditions only to find that failed preconditions somehow opended a 
> security hole....
>
Security is an aspect that cuts across functional boundaries.  If you 
have a
need for security, you must ensure the absolute absence of undefined
behaviour from your system (which is why I wince at books on "writing
secure code" which have basic examples with undefined behaviour 
lurking).

It is reasonable for a system to clarify what "undefined behaviour" 
means
for it: for example, an OS kernel doesn't define exactly the behaviour 
of
ever invalid call, and some may do slightly odd things to your process 
--
but the OS still provides guarantees such as that a user space process
will not be able to disrupt other processes unless permissions so allow,
and will not be allowed to cause a kernel panic.  Similarly server
programs which respond to network requests might not define the
result of every erroneous request, but in most deployment situations
they absolutely should guarantee that they won't open up security holes
and that they won't crash because of request received over the network.

Even within such a system there *can* be library calls that *don't* 
make such
guarantees for bad calls, because they assume that they can only be 
called
from other trusted code.  Clearly it's possible to increase the 
robustness of
security by making fewer such assumptions, but beyond a certain point 
the
added complexity itself tends to lead to other security problems.

-- James