[Ipopt] Numerical differences between runs

[Andreas Waechter]

Hi Gwyn,

In principle, the Ipopt code is deterministic, and results should be 100% 
reproducible.  This assumes that we are talking about running the same 
problem with exactly the same binary input repeatedly.

Possible causes for non-repeatability that I can imagine are:

- A memory bug (in your code or Ipopt code).  If uninitialized memory is 
used, the result is unpredictable.  Can you run your code through a memory 
checker?

- What linear solver are you using?  And what BLAS are you using?  I'm not 
sure what is in the Windows DLL binaries, but my guess is that it is MUMPS 
and netlib BLAS (I'm not compiling the DLLs, and I forgot what Marcel puts 
in there).  This should be deterministic code.  But maybe you are using a 
different linear solver, or some different BLAS (such as the MKL)?  When 
you compare the output with print level J_MOREVECTOR, are the first 
differences observed in the solution of the linear solver (you write resid 
and rhs, but which one is the first, or is it in a "SOL" component)?  If 
so, it probably depends on what linear solver (and underlying BLAS) you 
are using to see how that could be made reproducible.  Also, I assume that 
you are not running things in parallel/multi-threaded.

- I assume that you are not using any Ipopt warmstart (that would not 
guarantee repeatability). And are you using always the same 
IpoptApplication, or a new one every time?  Maybe the non-repeatability 
goes away if you always use a new IpoptApplication?  (If that is the case, 
it means that Ipopt carries some unintended information from one solve to 
the next, and I would like to know about that).

In general, for difficult problems, I have often observed a different 
number of iterations (or sometimes success vs. failure) when compiling the 
code with different compiler options, or when changing the operating 
system, particularly for difficult problems.  Unfortunately, it is in the 
nature of the floating point operations that the solution of the linear 
systems might differ drastically if those linear systems are 
ill-conditioned, and some floating point operations are done in a 
different order (which may be mathematically equivalent in precise 
arithmetic).

Not sure if this helps, but it might point you into the right direction.

Regards,

Andreas


On Wed, 24 Feb 2010, Gwyn Roberts wrote:

> We are relatively new users of IpOpt and have been gaining
experience over the last several months with a small to medium scale (2-4K 
variables/equations, 8-10K NZ (50% nonlinear)) nonlinear model 
application. We are using the version 3.8.1 release binariesfor Windows 
(as a DLL). Our full scale models are to the point where they solve quite 
nicely and robustly for most cases. However, we are encountering 
unexplained differences in “trajectory” (e.g. number of iterations, 
final convergence tightness, etc.) from run to run. This is the nature of 
our main concern – repeatability of performance and results. For our 
full model runs, the differences are manifest as different behaviours that 
range from 1 or 2 extra iterations to some that near the point of closure 
end up tripling the number of iterations and therefore the times.

We find that the trajectory for a problem will change depending on the 
problem solved previously, or, if running only a single problem, will 
change depending on whether or not a file journal is added. From looking 
at J_MOREVECTOR level output, we see that the departure in trajectory 
seems to take a while to take hold (3-8 iterations). The first observable 
behaviour is around the Barrier Parameter Update step and the max norm 
calculations related to the underlying calculations there (we have settled 
on adaptive strategy, but see the issues with monotone as well). There are 
numerical differences in the 8-15th significant figures of the resid and 
rhs vectors displayed at those journal levels. These appear ultimately to 
affect the relevant norms in significant ways that may alter the behaviour 
of the barrier slightly.

Can you comment on whether this is behaviour you have seen or experienced 
yourself (i.e. this is known behaviour, issue)? If you have any thoughts 
or guidance (or code changes) that might help us to eliminate the floating 
point variability, we would greatly appreciate your help. We have not yet 
taken the step of doing a full build of IpOpt ourselves, but have the 
ability to do so. We would wait to hear your reaction and thoughts before 
taking that on.

Thank you for your time and assistance.

Gwyn