From stefan at math.hu-berlin.de Fri Apr 1 05:43:04 2011 From: stefan at math.hu-berlin.de (Stefan Vigerske) Date: Fri, 01 Apr 2011 11:43:04 +0200 Subject: [Osi-managers] Osi unittest Message-ID: <4D959E28.7080406@math.hu-berlin.de> Hi, I recently commited some changes to the Osi unittest which hopefully make it easier to figure out where a test was failing. https://projects.coin-or.org/Osi/changeset/1676/trunk There is now a global instantiation OsiUnitTest::outcomes of the new class TestOutcomes which is used to collect outcomes of tests. Each outcome consists of - a component name (in most cases the Osi-interface name), - a test name - a test condition that was checked, or a test message - a severity level: - NOTE is to pass messages, e.g., about skipped tests - PASSED is for tests have been passed successfully - WARNING is for warnings - ERROR is for errors - a flag "expected", which make not much sense for PASSED, but for the others one can indicate whether, e.g., the failure of a test was expected (sometimes used for OsiVol) - the name of the file where the test is coded - the linenumber where the test is coded Currently only the tests in OsiCommonTests and the OsiGlpkSolverInterfaceTest make use of this object, other OsiXyzSolverInterfaceTest still need to be adapted. There are some macros that intend to make life easier. One set of macros is for testing whether a condition is true and generating adding the corresponding TestOutcome to the TestOutcomes object. E.g., if before you had if( a != b ) { failureMessage(solverName, "a should be b"); ++errCnt; } then now you can write OSIUNITTEST_ASSERT_ERROR(a == b, ++errCnt, solverName, "test a and b"); This addes a TestOutcome with the following entries: - component name is what is stored in the string solverName (alternatively, you can pass a OsiSolverInterface object here) - test name is "test a and b" - test condition is "a == b" - severity level is PASSED if a == b, and ERROR otherwise - expected is not set - for filename __FILE__ is used - for linenumber __LINE__ is used Further, the failureMessage function is called and the code given in the second argument (++errCnt) is executed. The errCnt is actually redundant now, but kept for backward compatibility. There are similar macros like - OSIUNITTEST_ASSERT_WARNING to specify if severity should be WARNING if the test fails - OSIUNITTEST_CATCH_SEVERITY_EXPECTED with additional arguments to specify the severity and "expected" flag The second set of macros is to test whether a code snippet produces an exception. E.g., if before you had something like try { si.initialSolve(); } catch (CoinError& e) { std::string errmsg; errmsg = "initialSolve threw CoinError: " + e.message(); failureMessage(*si, "test initial solve", errmsg.c_str()); ++errCnt; return errCnt; } then now you can write OSIUNITTEST_CATCH_ERROR(si.initialSolve(), return ++errCnt, si, "test initial solve"); This adds a TestOutcome with the following entries: - component name is obtained via si.getStrParam(OsiSolverName); - test name is "test initial solve" - test condition is - "si.initialSolve() threw CoinError: "+e.message() if a CoinError was thrown - "si.initialSolve() throw unknown exception" if some other exception was thrown - "si.initialSolve() did not threw exception" if no exception was thrown - severity level is PASSED or ERROR - expected is not set - filename is __FILE__ - linenumber is __LINE__ Additionally, if an exception is catched, a failure message is printed and the code "return ++errCnt;" is printed. To print the outcomes of the unittest, one can do OsiUnitTest::outcomes.print(); Per default, this currently prints only collected notes, warnings, and errors and a statistic on number of outcomes for each severity level. E.g., for Osi unittest if Glpk is available I get NOTE glpk testSimplexAPI (expected) skipped test ../../../../Osi/src/OsiCommonTest/OsiSimplexAPITest.cpp:740 WARNING glpk testHintParam: hint 1 sense 1 strength 3 if (si->setHintParam(key,sense,strength)) { ret = (si->getHintParam(key,post_sense,post_strength) == true) && (post_strength == strength) && (post_sense == sense); } threw CoinError: glpk does not support exclusive use of dual simplex ../../../../Osi/src/OsiCommonTest/OsiSolverInterfaceTest.cpp:1996 NOTE glpk testDualRays: getDualRays is implemented hasGetDualRays ../../../../Osi/src/OsiCommonTest/OsiSolverInterfaceTest.cpp:3633 Severity NOTE : 2 thereof expected: 1 Severity PASSED : 1507 thereof expected: 0 Severity WARNING : 4 thereof expected: 0 Severity ERROR : 0 thereof expected: 0 (it looks better on a wide screen :-)). To get the number of errors for one severity level, you can do int nerrors; int nerrors_expected; OsiUnitTest::outcomes.getCountBySeverity(OsiUnitTest::TestOutcome::ERROR, nerrors, nerrors_expected); Currently, unitTest reports success iff nerrors == nerrors_expected. Further, there is a new global variable OsiUnitTest::verbosity to specify verbosity of unittest output. Default (=0) is to print only minimal output, i.e., messages on failed tests (this is not fully implemented yet). Level 1 is to print more information on errors. And Level 2 prints also information on passed tests, also in the output of TestOutcomes::print(). The verbosity level can be set via an additional parameter -verbosity to the unittest executable. I replaced most assert(...) in the common unit tests by things like OSIUNITTEST_ASSERT_ERROR(.., ++errCnt; return, ...); As a consequence, I took out the #ifdef NDEBUG #undef NDEBUG #endif. At least to me this did not look like clean code. Stefan From stefan at math.hu-berlin.de Fri Apr 1 05:45:52 2011 From: stefan at math.hu-berlin.de (Stefan Vigerske) Date: Fri, 01 Apr 2011 11:45:52 +0200 Subject: [Osi-managers] CHANGELOG Message-ID: <4D959ED0.10108@math.hu-berlin.de> Hi, I started a CHANGELOG file in Osi with the aim to make announcements of new Osi versions easier for the maintainer. It would be nice if people could comment on changes in the API, new functionalities, fixed bugs, and when new versions are created there. Stefan