[Ipopt] Difficulties with Ipopt 3.9.1 on Linux (Ubuntu 10.04)

[John Pye]

Hi Stefan

Thanks very much for your reply to my long email. Just some followups...

Stefan Vigerske wrote:
> Hi,
>
> since your email is mostly about the build system, but nothing
> particular about Ipopt, let's move over to the buildtools mailing list.
>
>   
>> I am trying to compile my own copy of Ipopt 3.9.1 on Ubuntu 10.04, which 
>> only offers a possibly-faulty* copy of 3.8.1 in its repositories.
>>
>> I have downloaded the tarball and unpacked it, then in ThirdParty/MUMPS, 
>> I successfully ran 'get.Mumps'.
>>
>> I checked that I have libblas-dev and liblapack-dev installed. I checked 
>> that I had no libmumps* package installed.
>>
>> Next I ran
>>
>> F77=gfortran ./configure --prefix=/usr
>> make -j2
>> make DESTDIR=/home/john/ipopt-inst install
>>
>> Firstly, the above does not automatically detect the preinstalled BLAS 
>> and LAPACK, which would be nice if possible.
>>     
>
> This should work. configure tries whether -lblas and -llapack works. If
> it doesn't, please open a BuildTools or Ipopt ticket, with the
> config.log from the Ipopt subdirectory attached.
>   

I have reported the bug, but I don't have a copy of the config.log at
this stage.
https://projects.coin-or.org/Ipopt/ticket/142

Perhaps it would be worth raising this with the person who did your
packaging of Ipopt 3.8.1 for Ubuntu, as they will know if any special
tricks were required specifically on this distro.

>   
>> I added --with-*-lib 
>> options specifying the paths to the BLAS and LAPACK .so files.
>>
>> After this, I received some errors at the install stage, relating to 
>> non-existence of libcoinhsl.so. I don't want to support HSL, because I 
>> am trying to create a distributable package, so I need a clean way of 
>> suppressing all actions relating to HSL. I struck upon specifying the 
>> path to a nonexistent HSL library as a way to do that.
>>     
>
> Even if you do not provide any HSL code, there will still be an empty
> libcoinhsl created in ThirdParty/HSL. For some technical reasons, this
> is more convenient then omitting the hsl libraries if none of the HSL
> codes are provided.
>   

I think that the preferred solution here would be for IPOPT to dlopen
whatever shared libraries it needs. You can then hardcode where you
expect those files to be, and if they're not there, you can fail
gracefully and move on. No need for dummy files, and no need for tricky
environment settings to tell Linux where the libraries are located.

>> Even with the HSL workaround, I still found some problems at the 
>> 'install' step relating some some issue with pkgconfig failing to detect 
>> *.pc files in the local directory. So I disabled pkg-config.
>>     
>
> One guess is that you did not do a make install in the base directory,
> but only in the build directory of Ipopt. There is one step at the very
> end of make install of Ipopt where it is assumed that all
> .pc-dependencies of Ipopt are installed.
>   

I have not been using any separate 'build' directory, I've just done
everything from the top-level ~/Ipopt-3.9.1 folder.

>> So now, I ended up with:
>>
>> F77=gfortran ./configure --prefix=/usr --enable-static --enable-shared 
>> --with-blas-lib=/usr/lib/libblas.so 
>> --with-lapack-lib=/usr/lib/liblapack.so 
>> --with-hsl-lib=/usr/lib/libcoinhsl.so --enable-pkg-config=no
>> make -j2
>> make DESTDIR=/home/john/ipopt-inst install
>>
>> A comment: it would be very helpful if, as with the SUNDIALS solver 
>> suite, the ./configure script could give a succinct summary at the end 
>> showing the components that are going to be supported in the resulting 
>> build. I am sure that the way I have suppressed HSL, for example, is not 
>> the correct way.
>>
>> Another comment: I note that the default location of files in /usr/share 
>> has changed (I believe) from /usr/share/doc/coin/Ipopt to 
>> /usr/share/coin/doc/Ipopt. This arrangement of the file heirarchy is 
>> non-standard compared to how most packages are organised on Linux, in my 
>> understanding. Package-specific documentation is normally all held 
>> underneath /usr/share/doc/packagename.
>>     
>
> The default location was changed because some projects also install data
> that is not documentation, so everything related to coin can now be
> found under /usr/share/coin, like many other /usr/share/xxx's.
>   

I guess that it OK, but it just happens that in this case there was no
documentation.

>   
>> I note that even with the 'enable-pkg-config=no' command added to my 
>> ./configure command, the resulting 'make install' still installs a file 
>> ~/ipopt-inst/usr/lib/pkgconfig/ipopt.pc, and additionally, the 'Libs:' 
>> line in that file contains a reference to /usr/lib/libcoinhsl.so', which 
>> doesn't exist and will certainly cause problems down the track (again, 
>> what is the preferred way to safely disable HSL support?). But at least 
>> this way the installation completes.
>>     
>
> The .pc files are always created, and are even parsed during configure,
> if pkg-config is disabled.
> Disabling pkg-config just disables the use of pkg-config.
>   

Even though I disabled pkg-config, the resulting .pc files are still
installed, furthermore. Maybe it's not clear whether this option means
"I won't use pkg-config to located my dependencies" or "I won't install
files that can be used by pkg-config".

>> I'm not sure, but I think these things have arisen in Ipopt 3.9.1, 
>> perhaps as a result of the transition to the new COIN Buildtools system 
>> -- is that right?
>>     
>
> Yes, part's of. A libcoinhsl was always created, new is that it is
> installed. System Blas and Lapack have always been checked for.
>   

Note above comment about the idea of dlopening libcoinhsl instead of
linking to it in a non-standard directory.

>> I believe that I had a build script for 3.9.1 that 
>> worked fine with just './configure --prefix=/usr'.
>>
>> It would be really helpful if the Ipopt build process could do a little 
>> more to support default system configurations such as the one I have 
>> here. If I have libblas on my system, in /usr/lib, for example, it would 
>> be good if the default was to use that, rather than to fail because 
>> ThirdParty/Blas/get.Blas has not been run. Ditto LAPACK.
>>     
>
> As said, please open a ticket for this. libblas should be recognized.
>   

Done

>   
>> Finally, is there any need for the default installation of all these 
>> libraries to be in subdirectories of /usr/lib? Is there a simple way to 
>> make them all default to /usr/lib, instead of /usr/lib/coin and 
>> /usr/lib/coin/ThirdParty?
>>     
>
> The idea was not to clutter up /usr/lib with a bunch of coin and
> thirdparty libraries. The reason for the extra ThirdParty subdirectory
> was to distinguish coin projects from third party projects.
> It's like other larger projects that have there own subdirectories in
> /usr/lib.
> Unfortunately, I did not see how to change the default for libdir in
> configure. I had liked it to be $prefix/lib/coin, so a user could change
> this. Instead, the default for libdir is still $prefix/lib, and
> everything get's installed into $libdir/coin and  $libdir/coin/ThirdParty.
>   

It would be really great to work out how to change this so that Ipopt
3.9.1 can be conformed to the Linux Way of Doing Things. Need something like

./configure --prefix=/usr --libdir=/usr/lib --libdirthirdparty=/usr/lib

I think that Linux puts all that stuff in /usr/lib for really good
reasons. My machine has 3700 files there. Mostly it makes loading of
plugins and such things much simpler, because they always know where to
get their dependencies; there is no need for rpath or environment
variables to keep everything straight. If I have my ASCEND plugins in
/usr/lib/ascend, and one of those plugins links against Ipopt, I now
need to set LD_LIBRARY_PATH to
/usr/lib/ascend:/usr/lib/coin:/usr/lib/coin/ThirdParty. That's
difficult! I can managed the first one using my internal configuration
scripts etc, but the last two would require me to sniff the contents of
pkg-config ipopt --libs, which is very cumbersome. Having system
libraries in /usr/lib makes life so much simpler for the users of your
library, and greatly simplifies package management.

A quick look at the subdirectories of /usr/lib shows that deeper-nested
files are libraries usable only by specific software, for example
/usr/lib/python2.6 contains some shared libraries that are intended only
for use by Python itself. The Python shared library, mind you, which is
used by other people when they want to link to Python, is still in
/usr/lib/libpython2.6.so. That goes back to the 'dlopen' comment
mentioned earlier; Python maintains internally the location of its
'extension' shared libraries, and knows how to load them via dlopen, so
no special environment path is needed.

The rpath option, but that is discouraged in Debian
http://www.eyrie.org/~eagle/notes/rpath.html
http://wiki.debian.org/RpathIssue

I believe that changing ld.so.conf is discouraged in Debian, but I
couldn't find any special mention of that idea. Certainly it is very
easily avoided in this case by putting all the files in /usr/lib.

> For the Ubuntu specific questions, I have no answer.
>
> Stefan
>   

Thanks again,

Cheers
JP


-- 
Dr John Pye
Dept of Engineering
Australian National University