[Coin-ipopt] Memory Allocation.

[Andreas Waechter]

Hi Eugene,

A few ideas/thoughts come to my mind regarding your message:

1. I assume that you compiled the code in 64bit mode?  (Otherwise I think 
you couldn't access the 6.5 gigs anyway)

2. MA27 needs more memory if the fill-in during the factorization becomes 
really large.  Unfortunately, MA27, as a Fortran 77 code, does not do any 
memory allocation internally, so one has to give it work space in advance. 
Here, Ipopt guesses the amount of workspace based on some sizes, such as 
the number of non-zeros in the matrix, and uses the options 
ma27_liw_init_factor and ma27_la_init_factor to decide by how much to 
overestimate the amount of memory.  Later, when MA27 runs out of memory 
(this is the iflag=-4 message), Ipopt increases the amount of memory for 
MA27 by the factor ma27_meminc_factor.  The defaults are:

ma27_liw_init_factor 5.0
ma27_la_init_factor 5.0
ma27_meminc_factor 10.0

The last default option might be too aggressive, since once MA27 runs out 
of memory, it might easily go over the amount of memory available.

You could try something like the following:

ma27_liw_init_factor 100.0
ma27_la_init_factor 100.0
ma27_meminc_factor 2.0

which will allocate a lot of memory from the beginning (maybe 100 is too 
much and you will run out of memory from the beginning), but the increases 
later are more conservative.

3. If you want, you could try to use another linear solver (such as 
Pardiso or WSMP).  My own practical experience with those is still 
somewhat limited, but it seems to me that Pardiso has better ordering 
algorithms than MA27 (MA27 is more than 20 years old...) and can perform 
better for large problems.  However, it might not be as robust for 
ill-conditioned problems.  But, if MA27 doesn't work for you, I suggest 
you try to use Pardiso (it's free if you are at a university)

4. Another observation is that MA27 runs out of memory already before the 
first iteration.  This means, that it runs out of memory when it is trying 
to compute the initial values for the constraint multipliers (these are 
usually initialized as the least-square multipliers for the dual 
infeasibility).  Maybe the linear system for this initial computation is 
really badly conditioned.  In that case, you can just skip the 
least-square computation (and the algorithm will initialized the 
multipliers with zero).  In order to do this, you need to set the option

constr_mult_init_max 0

5. Finally, the message that I always preach:  Try to formlate your 
optimization problems as well scaled as possible, i.e., try to find a 
scaling for the variables and constraints, so that the "typical" values of 
non-zero first derivatvies is about 1 or so.  Also, try to choose a good 
starting point (e.g., in case your variables are all initalized as zero, 
many entries in the constraint Jacobian might all be zero, which would not 
be small otherwise).  Playing with that (including useing some random 
values around typical values for the individual entries) might in general 
be a good idea.

Well, I hope at least one of those suggestions helps a little bit...

Andreas

On Sat, 17 Feb 2007 deprince at uchicago.edu wrote:

> Hello,
>
> I have been using Ipopt for some time now with much success,
> benchmarking my own calculations against existing canned
> packages.  My problem has come up recently when the number of
> non-zeros in my constraint jacobian hits 1 million.  I first
> received errors regarding the size of the workspace for the
> ma27 solver:
>
> Number of nonzeros in equality constraint Jacobian...:  1229184
> Number of nonzeros in inequality constraint Jacobian.:        0
> Number of nonzeros in Lagrangian Hessian.............:        0
>
> MA27BD returned iflag=-4 and requires more memory.
> Increase liw from 12321685 to 123216850 and la from 1234617
> to 12377700 and factorize again.
> MA27BD returned iflag=-4 and requires more memory.
> Increase liw from 123216850 to 1232168500 and la from
> 12377700 to 126782410 and factorize again.
>
> EXIT: Not enough memory.
>
> I bypass the first errors using the ma27_liw_init_factor and
> ma27_la_init_factor options, but I still recieve EXIT: Not
> enough memory.
>
> When I enter the solver, there are about 6.5 gigs of memory
> available, so I'm not sure why the program crashes.  Could the
> problem be with the memory allocation for ma27?  I'm not
> really familiar with the way FORTRAN allocates memory, so I
> don't know if this is the case.  If anyone has any ideas, I
> would really appreciate them.
>
> Thanks in advance,
> Eugene
> _______________________________________________
> Coin-ipopt mailing list
> Coin-ipopt at list.coin-or.org
> http://list.coin-or.org/mailman/listinfo/coin-ipopt
>