[Ipopt] MPI Issue in IPOPT 3.11.7 / Mumps

[Dominique Orban]

Thanks Damien. Your suggestion solves the problem of the C test, and
all tests now pass. I got in touch with Mumps guys to see what they
suggest regarding MPI_Initialized() and MPI_Finalized() in the
sequential scenario.

Cheers,
Dominique


On Thu, Feb 20, 2014 at 9:50 PM, Damien <damien at khubla.com> wrote:
> Apologies for my previous reply about sequential, I didn't read it
> correctly.  What you're seeing is the result of MPI_Init() being called
> twice without an MPI_Finalize() between the calls.  I don't have the Ipopt
> code in front of me right now, but: If there's an exit path from the
> parallel MPI MUMPS that doesn't call MPI_Finalize() after a completed Ipopt
> solve, when you warm-start Ipopt, MPI_Init gets called again and MPI aborts.
>
> Making Ipopt fully MPI compliant at the top level of Ipopt is not the right
> software engineering answer.  MUMPS is the MPI solver, so the responsibility
> should be in the MUMPS wrapper.  Keeping MPI alive or not should be a
> MUMPS-specific option.
>
> To check if MPI is initialised or not:
>
>  int is_initialized = 0;
>  MPI_Initialized(&is_initialized);
>  if(!is_initialized)
>  {
>     MPI_Init(&argc, &argv);
>  }
>
>
> and to check if it's finalised:
>
>  int is_finalized = 0;
>  MPI_Finalized(&is_finalized);
>  if(!is_finalized)
>  {
>     MPI_Finalize();
>  }
>
>
>
> Just putting these in will likely fix most of the problems, but if there's a
> path to finish a solve without calling MPI_Finalize, that's a bug that needs
> fixing.
>
> Also note that the MUMPS fake MPI for sequential (in the MUMPS libseq
> directory) doesn't have an implementation for MPI_Initialized or
> MPI_Finalized and those stubs need to be added to compile the code above.
>
> Cheers,
>
> Damien
>
>
> On 2014-02-20 9:20 PM, Tony Kelman wrote:
>>
>> Thanks Dominique!
>>
>> As I mentioned in the discussion that brought this up on Github
>> (https://github.com/Homebrew/homebrew-science/issues/656), I think one
>> possible approach might be to move the MPI_Init() and MPI_Finalize() out of
>> the MumpsSolverInterface constructor/destructors, and put them in the Ipopt
>> solver object's constructor/destructor instead. We'd need to do some
>> ifdef'ing to skip calling MPI_* if we don't have access to the commonly-used
>> fake MPI headers from sequential Mumps.
>>
>> However I'm not so sure if that would completely solve the problem, and I
>> wanted to get others' opinions here who may have better ideas or have come
>> across this before.
>>
>> -Tony
>>
>> -----Original Message----- From: Dominique Orban
>> Sent: Thursday, February 20, 2014 8:04 PM
>> To: ipopt at list.coin-or.org
>> Subject: [Ipopt] MPI Issue in IPOPT 3.11.7 / Mumps
>>
>> Hi there,
>>
>> Building IPOPT 3.11.7 with clang/clang++ or gcc/g++ on OSX 10.8.5 against
>> MUMPS 4.10.0 and OpenMPI 1.7.4 results in the C test hs071_c hanging forever
>> or crashing.
>>
>> There appears to be a problem with the MPI setup in the Mumps interface.
>> The C test solves the problem twice (the second solve with warm starting).
>> But this means that MPI_Init() is called twice and that leads to an error:
>> https://gist.github.com/dpo/9120810
>>
>> A user suggested that IPOPT shouldn't call MPI_Init() at all and proposes
>> a patch (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=736842) but
>> applying this patch results in a situation where MPI_Init() is never called:
>>
>> The MPI_Comm_f2c() function was called before MPI_INIT was invoked.
>>
>> Both my C++ and MPI knowledge are extremely limited. Somehow, MPI_Init()
>> is called again when re-solving and/or MPI_Finalize() is never called. Or
>> something else is going on... I got in touch with the Mumps guys to see if
>> they have any advice.
>>
>> Cheers,
>>
>> Dominique
>>
>>
>> _______________________________________________
>> Ipopt mailing list
>> Ipopt at list.coin-or.org
>> http://list.coin-or.org/mailman/listinfo/ipopt
>> _______________________________________________
>> Ipopt mailing list
>> Ipopt at list.coin-or.org
>> http://list.coin-or.org/mailman/listinfo/ipopt
>
>
> _______________________________________________
> Ipopt mailing list
> Ipopt at list.coin-or.org
> http://list.coin-or.org/mailman/listinfo/ipopt