[Osi-managers] Osi unittest

[Stefan Vigerske]

Hi,

I recently commited some changes to the Osi unittest which hopefully 
make it easier to figure out where a test was failing.
https://projects.coin-or.org/Osi/changeset/1676/trunk

There is now a global instantiation OsiUnitTest::outcomes of the new 
class TestOutcomes which is used to collect outcomes of tests. Each 
outcome consists of
- a component name (in most cases the Osi-interface name),
- a test name
- a test condition that was checked, or a test message
- a severity level:
   - NOTE is to pass messages, e.g., about skipped tests
   - PASSED is for tests have been passed successfully
   - WARNING is for warnings
   - ERROR is for errors
- a flag "expected", which make not much sense for PASSED, but for the
   others one can indicate whether, e.g., the failure of a test was
   expected (sometimes used for OsiVol)
- the name of the file where the test is coded
- the linenumber where the test is coded

Currently only the tests in OsiCommonTests and the 
OsiGlpkSolverInterfaceTest make use of this object, other 
OsiXyzSolverInterfaceTest still need to be adapted.

There are some macros that intend to make life easier.
One set of macros is for testing whether a condition is true and 
generating adding the corresponding TestOutcome to the TestOutcomes 
object. E.g., if before you had
   if( a != b ) {
     failureMessage(solverName, "a should be b");
     ++errCnt;
   }
then now you can write
  OSIUNITTEST_ASSERT_ERROR(a == b, ++errCnt, solverName, "test a and b");

This addes a TestOutcome with the following entries:
- component name is what is stored in the string solverName
   (alternatively, you can pass a OsiSolverInterface object here)
- test name is "test a and b"
- test condition is "a == b"
- severity level is PASSED if a == b, and ERROR otherwise
- expected is not set
- for filename __FILE__ is used
- for linenumber __LINE__ is used
Further, the failureMessage function is called and the code given in the 
second argument (++errCnt) is executed.
The errCnt is actually redundant now, but kept for backward compatibility.

There are similar macros like
- OSIUNITTEST_ASSERT_WARNING to specify if severity should be WARNING
   if the test fails
- OSIUNITTEST_CATCH_SEVERITY_EXPECTED with additional arguments to
   specify the severity and "expected" flag

The second set of macros is to test whether a code snippet produces an 
exception. E.g., if before you had something like
   try {
     si.initialSolve();
   } catch (CoinError& e) {
     std::string errmsg;
     errmsg = "initialSolve threw CoinError: " + e.message();
     failureMessage(*si, "test initial solve", errmsg.c_str());
     ++errCnt;
     return errCnt;
   }
then now you can write
   OSIUNITTEST_CATCH_ERROR(si.initialSolve(), return ++errCnt, si,
                                          "test initial solve");

This adds a TestOutcome with the following entries:
- component name is obtained via si.getStrParam(OsiSolverName);
- test name is "test initial solve"
- test condition is
   - "si.initialSolve() threw CoinError: "+e.message() if a CoinError
      was thrown
   - "si.initialSolve() throw unknown exception" if some other exception
      was thrown
   - "si.initialSolve() did not threw exception" if no exception was
      thrown
- severity level is PASSED or ERROR
- expected is not set
- filename is __FILE__
- linenumber is __LINE__
Additionally, if an exception is catched, a failure message is printed 
and the code "return ++errCnt;" is printed.

To print the outcomes of the unittest, one can do
   OsiUnitTest::outcomes.print();
Per default, this currently prints only collected notes, warnings, and 
errors and a statistic on number of outcomes for each severity level.

E.g., for Osi unittest if Glpk is available I get
NOTE      glpk      testSimplexAPI
  (expected)         skipped test
 
../../../../Osi/src/OsiCommonTest/OsiSimplexAPITest.cpp:740
WARNING   glpk      testHintParam: hint 1 sense 1 strength 3
                     if (si->setHintParam(key,sense,strength)) { ret = 
(si->getHintParam(key,post_sense,post_strength) == true) && 
(post_strength == strength) && (post_sense == sense); } threw CoinError: 
glpk does not support exclusive use of dual simplex
 
../../../../Osi/src/OsiCommonTest/OsiSolverInterfaceTest.cpp:1996
NOTE      glpk      testDualRays: getDualRays is implemented
                     hasGetDualRays
 
../../../../Osi/src/OsiCommonTest/OsiSolverInterfaceTest.cpp:3633

Severity NOTE      :    2  thereof expected:    1
Severity PASSED    : 1507  thereof expected:    0
Severity WARNING   :    4  thereof expected:    0
Severity ERROR     :    0  thereof expected:    0

(it looks better on a wide screen :-)).

To get the number of errors for one severity level, you can do
int nerrors;
int nerrors_expected; 
OsiUnitTest::outcomes.getCountBySeverity(OsiUnitTest::TestOutcome::ERROR, nerrors, 
nerrors_expected);

Currently, unitTest reports success iff nerrors == nerrors_expected.


Further, there is a new global variable OsiUnitTest::verbosity to 
specify verbosity of unittest output.
Default (=0) is to print only minimal output, i.e., messages on failed 
tests (this is not fully implemented yet). Level 1 is to print more 
information on errors. And Level 2 prints also information on passed 
tests, also in the output of TestOutcomes::print().
The verbosity level can be set via an additional parameter -verbosity to 
the unittest executable.


I replaced most assert(...) in the common unit tests by things like 
OSIUNITTEST_ASSERT_ERROR(.., ++errCnt; return, ...);

As a consequence, I took out the #ifdef NDEBUG #undef NDEBUG #endif.
At least to me this did not look like clean code.

Stefan