[BuildTools] [Ipopt] Difficulties with Ipopt 3.9.1 on Linux (Ubuntu 10.04)

Thu Dec 23 07:17:40 EST 2010

Hi,

> I have reported the bug, but I don't have a copy of the config.log at
> this stage.
> https://projects.coin-or.org/Ipopt/ticket/142

It would really be good to have some config.log file.

> Perhaps it would be worth raising this with the person who did your
> packaging of Ipopt 3.8.1 for Ubuntu, as they will know if any special
> tricks were required specifically on this distro.

I don't know who is doing this...

>>> I added --with-*-lib 
>>> options specifying the paths to the BLAS and LAPACK .so files.
>>>
>>> After this, I received some errors at the install stage, relating to 
>>> non-existence of libcoinhsl.so. I don't want to support HSL, because I 
>>> am trying to create a distributable package, so I need a clean way of 
>>> suppressing all actions relating to HSL. I struck upon specifying the 
>>> path to a nonexistent HSL library as a way to do that.
>>>     
>>
>> Even if you do not provide any HSL code, there will still be an empty
>> libcoinhsl created in ThirdParty/HSL. For some technical reasons, this
>> is more convenient then omitting the hsl libraries if none of the HSL
>> codes are provided.
>>   
> 
> I think that the preferred solution here would be for IPOPT to dlopen
> whatever shared libraries it needs. You can then hardcode where you
> expect those files to be, and if they're not there, you can fail
> gracefully and move on. No need for dummy files, and no need for tricky
> environment settings to tell Linux where the libraries are located.

Yes, that would be nice, but implementing this for all libraries will
not be done. Also dynamically loading something that is in C++ is more
involved, for the MS compiler one would need to mark every function that
can be dynamically loaded explictely, and dlopen does not exists on
every system.
However, for the routines from the HSL there is a mechanism in Ipopt to
load them dynamically at runtime.

>>> Even with the HSL workaround, I still found some problems at the 
>>> 'install' step relating some some issue with pkgconfig failing to detect 
>>> *.pc files in the local directory. So I disabled pkg-config.
>>>     
>>
>> One guess is that you did not do a make install in the base directory,
>> but only in the build directory of Ipopt. There is one step at the very
>> end of make install of Ipopt where it is assumed that all
>> .pc-dependencies of Ipopt are installed.
>>   
> 
> I have not been using any separate 'build' directory, I've just done
> everything from the top-level ~/Ipopt-3.9.1 folder.

That's fine. The question was if you called make install from
~/Ipopt-3.9.1 or from ~/Ipopt-3.9.1/Ipopt. The latter only works, if all
dedendencies have been installed before. The former ensures this.

>>> So now, I ended up with:
>>>
>>> F77=gfortran ./configure --prefix=/usr --enable-static --enable-shared 
>>> --with-blas-lib=/usr/lib/libblas.so 
>>> --with-lapack-lib=/usr/lib/liblapack.so 
>>> --with-hsl-lib=/usr/lib/libcoinhsl.so --enable-pkg-config=no
>>> make -j2
>>> make DESTDIR=/home/john/ipopt-inst install
>>>
>>> A comment: it would be very helpful if, as with the SUNDIALS solver 
>>> suite, the ./configure script could give a succinct summary at the end 
>>> showing the components that are going to be supported in the resulting 
>>> build. I am sure that the way I have suppressed HSL, for example, is not 
>>> the correct way.
>>>
>>> Another comment: I note that the default location of files in /usr/share 
>>> has changed (I believe) from /usr/share/doc/coin/Ipopt to 
>>> /usr/share/coin/doc/Ipopt. This arrangement of the file heirarchy is 
>>> non-standard compared to how most packages are organised on Linux, in my 
>>> understanding. Package-specific documentation is normally all held 
>>> underneath /usr/share/doc/packagename.
>>>     
>>
>> The default location was changed because some projects also install data
>> that is not documentation, so everything related to coin can now be
>> found under /usr/share/coin, like many other /usr/share/xxx's.
>>   
> 
> I guess that it OK, but it just happens that in this case there was no
> documentation.
> 
>>   
>>> I note that even with the 'enable-pkg-config=no' command added to my 
>>> ./configure command, the resulting 'make install' still installs a file 
>>> ~/ipopt-inst/usr/lib/pkgconfig/ipopt.pc, and additionally, the 'Libs:' 
>>> line in that file contains a reference to /usr/lib/libcoinhsl.so', which 
>>> doesn't exist and will certainly cause problems down the track (again, 
>>> what is the preferred way to safely disable HSL support?). But at least 
>>> this way the installation completes.
>>>     
>>
>> The .pc files are always created, and are even parsed during configure,
>> if pkg-config is disabled.
>> Disabling pkg-config just disables the use of pkg-config.
>>   
> 
> Even though I disabled pkg-config, the resulting .pc files are still
> installed, furthermore. Maybe it's not clear whether this option means
> "I won't use pkg-config to located my dependencies" or "I won't install
> files that can be used by pkg-config".

The documentation of --enable-pkg-config says "use pkg-config if
available", it does not says anything about .pc files.
Since we need these files in any case, i.e., also if pkg-config is
disabled, the idea was to also install them, in case someone may still
need them later.

>>> I'm not sure, but I think these things have arisen in Ipopt 3.9.1, 
>>> perhaps as a result of the transition to the new COIN Buildtools system 
>>> -- is that right?
>>>     
>>
>> Yes, part's of. A libcoinhsl was always created, new is that it is
>> installed. System Blas and Lapack have always been checked for.
>>   
> 
> Note above comment about the idea of dlopening libcoinhsl instead of
> linking to it in a non-standard directory.
> 
>>> I believe that I had a build script for 3.9.1 that 
>>> worked fine with just './configure --prefix=/usr'.
>>>
>>> It would be really helpful if the Ipopt build process could do a little 
>>> more to support default system configurations such as the one I have 
>>> here. If I have libblas on my system, in /usr/lib, for example, it would 
>>> be good if the default was to use that, rather than to fail because 
>>> ThirdParty/Blas/get.Blas has not been run. Ditto LAPACK.
>>>     
>>
>> As said, please open a ticket for this. libblas should be recognized.
>>   
> 
> Done
> 
>>   
>>> Finally, is there any need for the default installation of all these 
>>> libraries to be in subdirectories of /usr/lib? Is there a simple way to 
>>> make them all default to /usr/lib, instead of /usr/lib/coin and 
>>> /usr/lib/coin/ThirdParty?
>>>     
>>
>> The idea was not to clutter up /usr/lib with a bunch of coin and
>> thirdparty libraries. The reason for the extra ThirdParty subdirectory
>> was to distinguish coin projects from third party projects.
>> It's like other larger projects that have there own subdirectories in
>> /usr/lib.
>> Unfortunately, I did not see how to change the default for libdir in
>> configure. I had liked it to be $prefix/lib/coin, so a user could change
>> this. Instead, the default for libdir is still $prefix/lib, and
>> everything get's installed into $libdir/coin and  $libdir/coin/ThirdParty.
>>   
> 
> It would be really great to work out how to change this so that Ipopt
> 3.9.1 can be conformed to the Linux Way of Doing Things. Need something like
> 
> ./configure --prefix=/usr --libdir=/usr/lib --libdirthirdparty=/usr/lib
> 
> I think that Linux puts all that stuff in /usr/lib for really good
> reasons. My machine has 3700 files there. Mostly it makes loading of
> plugins and such things much simpler, because they always know where to
> get their dependencies; there is no need for rpath or environment
> variables to keep everything straight. If I have my ASCEND plugins in
> /usr/lib/ascend, and one of those plugins links against Ipopt, I now
> need to set LD_LIBRARY_PATH to
> /usr/lib/ascend:/usr/lib/coin:/usr/lib/coin/ThirdParty. That's
> difficult! I can managed the first one using my internal configuration
> scripts etc, but the last two would require me to sniff the contents of
> pkg-config ipopt --libs, which is very cumbersome. Having system
> libraries in /usr/lib makes life so much simpler for the users of your
> library, and greatly simplifies package management.
> 
> A quick look at the subdirectories of /usr/lib shows that deeper-nested
> files are libraries usable only by specific software, for example
> /usr/lib/python2.6 contains some shared libraries that are intended only
> for use by Python itself. The Python shared library, mind you, which is
> used by other people when they want to link to Python, is still in
> /usr/lib/libpython2.6.so. That goes back to the 'dlopen' comment
> mentioned earlier; Python maintains internally the location of its
> 'extension' shared libraries, and knows how to load them via dlopen, so
> no special environment path is needed.
> 
> The rpath option, but that is discouraged in Debian
> http://www.eyrie.org/~eagle/notes/rpath.html
> http://wiki.debian.org/RpathIssue
> 
> I believe that changing ld.so.conf is discouraged in Debian, but I
> couldn't find any special mention of that idea. Certainly it is very
> easily avoided in this case by putting all the files in /usr/lib.

We will think about this.

Stefan

> 
>> For the Ubuntu specific questions, I have no answer.
>>
>> Stefan
>>   
> 
> Thanks again,
> 
> Cheers
> JP
> 
> 

-- 
Stefan Vigerske
Humboldt University Berlin, Numerical Mathematics
http://www.math.hu-berlin.de/~stefan