[Os-project-managers] OSrL and unit tests

[R. Kipp Martin]

Hi Mike:

I meet with Gus and Jun next week. Let me get back to you after we meet. 
  What you are doing sounds like a good plan and we should somehow 
collaborate.

Cheers
> The test we have implemented is a test of the CMPL binary.
>
> The first step was to generate the MPS or OSiL files for a couple of
> selected CMPL problems using an older stable version of CMPL. The stout
> and/or stderr of some of these problems were also be saved in files. We
> checked all of these output files by hand. As a result we have for all
> of the selected CMPL files pairs of the inputs ( <problemname>.cmpl )
> and the specific outputs (<problemname>.comp.mps,
> <problemname>.comp.osil <problemname>.comp.stdout,
> <problemname>.comp.stderr>. For each pair we are sure that this
> combination is correct; free of (software) errors.
>
> Each new CMPL version has to generate a identical output for a specific
> input.
>
> We use two simple bash scripts to check whether a new CMPL version is
> able to generate the expected outputs.
>
> The script test-all in
> https://projects.coin-or.org/svn/Cmpl/releases/1.5.2/ is the test
> framework that executes all tests.
>
> # Test whether cmpl exist or not
> test -s ../bin/cmpl; r0=$?
> test -s ../bin/cmpl.exe; r1=$?
>
> if [ "$r0" = 0 -o "$r1" = 0 ]; then
> echo "model generating tests"
> ./test-cmpl -ca -m 01_Standardmodell.mps 01_Standardmodell.cmpl
> ./test-cmpl -ca -m 02_Standardmodell.mps 02_Standardmodell.cmpl
> ./test-cmpl -ff -ca -f%20.0f -m 03_Fibonacci.mps 03_Fibonacci.cmpl
> ./test-cmpl -ca -cd -m 04_Primzahlen.mps 04_Primzahlen.cmpl
> ./test-cmpl -ca -m 05_IntervallLinearisierung.mps
> 05_IntervallLinearisierung.cmpl
> ./test-cmpl -m 06_Variablenprodukte.A.mps 06_Variablenprodukte.A.cmpl
> ./test-cmpl -m 06_Variablenprodukte.B.mps 06_Variablenprodukte.B.cmpl
> ./test-cmpl -m 06_Variablenprodukte.C.mps 06_Variablenprodukte.C.cmpl
> ./test-cmpl -m 07_Mehrkriteriell.mps 07_Mehrkriteriell.cmpl
> ./test-cmpl -m 08_SummeNegiert.mps 08_SummeNegiert.cmpl
> ./test-cmpl -ca -m 09_Beispiel_Steglich.mps 09_Beispiel_Steglich.cmpl
> ./test-cmpl -m 10_Beispiel_Rost.mps 10_Beispiel_Rost.cmpl
> ./test-cmpl -ca -m 11_Beispiel_Roemer.mps 11_Beispiel_Roemer.cmpl
> ./test-cmpl -ff -m 11_Beispiel_Roemer2.mps 11_Beispiel_Roemer2.cmpl
> ./test-cmpl -cd -noOutput 12_Test_Loop.cmpl
> ./test-cmpl -cd -noOutput 13_Test_Set.cmpl
> ./test-cmpl -ff -m 14_Test_Summe.mps 14_Test_Summe.cmpl
> ./test-cmpl -x 15_Test_Osil.osil 15_Test_Osil.cmpl
> ./test-cmpl -x 16_diet.osil 16_diet.cmpl
> ./test-cmpl -ca -cd -noOutput 17_knapsack-min-presolved.cmpl
> ./test-cmpl -ca -cd -noOutput 18_max-negative-convex-function.cmpl
> ./test-cmpl -ca -x 19_quadratic-assignment.osil 19_quadratic-assignment.cmpl
> ./test-cmpl -noOutput 20_math-functions.cmpl
> ./test-cmpl -x 21_testcsvrand.osil 21_testcsvrand.cmpl
> echo "model generating and solver tests"
> ./test-cmpl 22_diet.cmpl
> ./test-cmpl -ca 23_production-mix.cmpl
>
> else
> echo "CMPL does not exist in ../bin"
> fi
>
> The script test-cmpl generates for an input ( <problemname>.cmpl ) the
> outputs (problemname>.mps, <problemname>.osil <problemname>.stdout,
> <problemname>.stderr) and compares the outputs with the original
> <problemname>.comp.mps, <problemname>.comp.osil and if not empty also
> with the <problemname>.comp.stdout and <problemname>.comp.stderr files.
> If there is no difference between the xxx.comp.xxx and the generated
> file than the test is successful.
>
> #!/bin/bash
>
> # cmpl test
> # Parameter: cmpl parameter
>
> # The cmpl file xxx.gen is to be compared with the following files:
> #xxx.comp.mps
> #xxx.comp.stdout (if not empty)
> #xxx.comp.stderr(if not empty)
>
> ERG=0
> function compfile() {
> NAME="$1"
> ERW="$2"
> FILE="$NAME.$ERW"
> if [ -f "$FILE" ]; then
> if [ "$FILE" -ot "$NAME.time" ]; then
> echo "$NAME: file $FILE was not newly created"
> echo "less $NAME.stderr"
> ERG=1
> exit $ERG
> fi
> fi
> test -s "$FILE"; FLA=$?
> test -s "$NAME.comp.$ERW"; FLB=$?
> if [ "$FLA" = 1 -o "$FLB" = 1 ]; then
> if [ "$FLA" = 0 ]; then
> echo "$NAME: file $FILE is not empty"
> echo "less $FILE"
> ERG=2
> exit $ERG
> elif [ "$FLB" = 0 ]; then
> echo "$NAME: file $FILE is empty"
> ERG=3
> exit $ERG
> fi
> else
> diff -b "$FILE" "$NAME.comp.$ERW" >/dev/null
> if [ $? != 0 ]; then
> echo "$NAME: file $FILE is different to the compared file"
> echo "diff $FILE $NAME.comp.$ERW | less"
> ERG=4
> exit $ERG
> fi
> fi
> }
>
> # file name w/o extension
> ANZPAR=$#
> eval "DATEI=\${$ANZPAR}"
> DATEI=${DATEI%.cmpl}
> #echo $DATEI
> # actual time
> touch "$DATEI.time"
>
> # call cmpl
> # creates a mps/Osil file and files for stdout and stderr
> ../bin/cmpl "$@" >"$DATEI.stdout" 2>"$DATEI.stderr"
>
> # tests for mps/Osil / stdout / stderr
> compfile "$DATEI" mps
> compfile "$DATEI" osil
> compfile "$DATEI" stdout
> compfile "$DATEI" stderr
>
> if [ "$ERG" = 0 ]; then
> echo "$DATEI : OK"
> fi
> exit $ERG
>
>
> With this test routine we are able to test the following cases:
> CMPL generates a MPS or Free MPS file (eg. ./test-cmpl -ca -m
> 01_Standardmodell.mps 01_Standardmodell.cmpl)
> CMPL is only used for calculation for that the results are to be
> displayed on stdout (eg. ./test-cmpl -ff -ca -f%20.0f -m
> 03_Fibonacci.mps 03_Fibonacci.cmpl)
> CMPL generates an error (e.g. /test-cmpl -cd -noOutput 13_Test_Set.cmpl)
> CMPL is used to generate an OSiL file and to solve it with the
> OSSolverservice (e.g. ./test-cmpl 22_diet.cmpl)
>
>
> An CMPL user that wants to compile CMPL can use this test using the
> command cmplMake -test.
>
>
> I can imagine that we are able to create such a test suite for OS with
> pairs of OSiL inputs and OSrL outputs (including if it makes sense also
> the stdout and stderr outputs).
>
> The only problem that I see at the moment is that OS writes an
> information about the version/release in the OSrL files. But I think
> this problem can be avoided by some string operations in the test scripts.
>
> Cheers
>
>
> Mike
>
>
>
>
>
> Am 13.10.2011 um 21:24 schrieb R. Kipp Martin:
>
>> Hi Mike:
>>
>> Sorry for the delay in responding.
>>
>>> We had similar problems with generated output files of CMPL. Our
>>> current approach is to compare selected CMPL output files (that are
>>> checked by hand) with the generated files of the compiled version.
>>> With this test it is possible to check whether the CMPL works and
>>> whether the output is correct.
>>> It works with a simple bash script that you can find using the
>>> following link:
>>> https://projects.coin-or.org/svn/Cmpl/releases/1.5.2/test/ .
>>>
>>> If it is interesting for you to have an additional test to the unit
>>> tests I can create such a test the OSrL files. This is only a
>>> suggestion but it would be a pleasure to give a contribution to your
>>> project.
>>
>> All help is gratefully accepted. However, I don't fully understand
>> what you are offering. Are you talking about C++ code, or OSiL files,
>> or OSrL files?
>>
>> Gus, Jun, and I just had a meeting and discussed your email. The
>> problem you had recently came from the OSCoinSolver.cpp generating bad
>> OSrL. We feel that testing OSrL files against the OSrL parser might
>> not be the most beneficial thing. Instead, having a broad spectrum of
>> OSiL test problems that test conditions like pivot limits exceeded,
>> infeasibility, etc. would be the best way to test the robustness of
>> the OS code. Is this what you are suggesting?
>>
>> Thanks
>>
>> --
>> Kipp Martin
>> Professor of Operations Research
>> and Computing Technology
>> Booth School of Business
>> University of Chicago
>> 5807 South Woodlawn Avenue
>> Chicago, IL 60637
>> 773-702-7456
>> kmartin at chicagobooth.edu <mailto:kmartin at chicagobooth.edu>
>> http://www.chicagobooth.edu/faculty/bio.aspx?person_id=12825325568
>> http://projects.coin-or.org/OS
>>
>> Sent without Blackberry, Droid, iPhone, or any other
>> wireless device.
>> --
>


-- 
Kipp Martin
Professor of Operations Research
and Computing Technology
Booth School of Business
University of Chicago
5807 South Woodlawn Avenue
Chicago, IL 60637
773-702-7456
kmartin at chicagobooth.edu
http://www.chicagobooth.edu/faculty/bio.aspx?person_id=12825325568
http://projects.coin-or.org/OS

Sent without Blackberry, Droid, iPhone, or any other
wireless device.
--