[BuildTools] [Coin-tlc] Help with build system

[Stefan Vigerske]

Hi,

Am 31.05.2010 22:24, schrieb Matěj Týč:
> Hello,
> thank you for your reply!
> On Sat, 2010-05-29 at 13:02 +0200, Stefan Vigerske wrote:
>> Hi,
>>
>> I don't know, the macro you suggest seem to assemble the compiler flags
>> at the time where configure is setup, but it should be at the time when
>> configure is executed.
> 
> The idea is that you will know the flags at autoreconf time and you may
> or may not use them at configure time.

But what if the dependency change? One still needs to rerun the
autotools for the higher project then. It is better than having to
adjust the flags by hand and rerun the autotools, though.

> You can use a similar logic with sourcing shell scripts, but m4sugar has
> those nice macros that make it easier :-)
> 
>> Also there is already a file that contains dependency and compiler flags
>> information: xxx-uninstalled.pc.
>>
>> Why not implement the following?
>> 1. If pkg-config is available, use it to collect compiler and linker
>> flags. Here it does not make a difference, if a project has been
>> installed or not yet.
> 
> Well, personally I wouldn't say that there is no difference. As I see
> it, if a project is installed, it has a .pc file in the "standard"
> directory, whereas the not-yet-installed project has it in its source
> tree. So as I see it, you have to invoke pkg-config in a different ways
> those two times.

We know where the uninstalled projects are, so their path is added to
the PKG_CONFIG_PATH. That's all.

>> 2. If pkg-config is not available, use gnulib to collect compiler and
>> linker flags of all installed dependencies. Since they are installed,
>> the compiler flags are in some way easy (-I<installdir>/include).
> 
> Yeah.
> 
>> 3. If the first two fails (no pkg-config available and dependencies are
>> not installed yet), try using gnulib to find linker flags in the build
>> directories of the dependencies and assemble compiler by parsing the
>> xxx-uninstalled.pc file (as already implemented).
> 
> I would say that we don't need to use the .pc file format since we don't
> need all of its features. I think that a primitive shell script (just
> two lines of assignments) that would be sourced in a "parent" script
> would do the job, too.

But we have the .pc files already. I do not understand why to create
another file with more or less the same information - only in a slightly
different format.

> And those macros I have provided probably would handle recursive
> dependencies too, like when you want to build library A, you need to
> access include files both from libraries B and C, where C depends only
> on B, not on A. I don't know how a big issue is this and whether it is
> not solved, too...
> However, I think that the future would be to get rid of situations like
> these anyway.

Yes, resolving dependencies is not in the fallback that is currently
implemented. The dependencies are stored in the .pc file, but there is
no script that resolves them (at configure time). (The idea was not to
reimplement pkg-config by shell scripts ;-).)
But yes, building against uninstalled projects is a situation that
should be avoided in the future anyway, so options 1 or 2 should always
be working.

>> If that fails, try parsing the xxx-uninstalled.pc file to assemble both
>> compiler and linker flags.
> 
> Moreover, if I understand correctly, if we intend to use a
> xxx-uninstalled.pc file, it means that we are dealing with our (=COIN)
> stuff that is not installed yet, right? 

Yes.

> If this is the case, it should be safe to assume that we are able to
> provide all *FLAGS correctly since we, the creators, know best. Which
> would mean that we can use all the info from the file
> (xxx-uninstalled.pc or whatever) right away (if I have understood
> correctly, now it is like take one piece, guess the rest and if it
> fails, only then take all).

You are right, that sounds like a useful simplification :-).

>> 4. Test the compiler and linker flags by attempting a link.
> 
> By the way, finding libraries with gnulib includes this step.

I know.

Stefan

> 
>>
>> Stefan
>>
>> Matěj Týč wrote:
>>> On Wed, 2010-05-19 at 23:38 +0200, Stefan Vigerske wrote: 
>>>> Hi,
>>>>
>>>> Matěj Týč wrote:
>>>>> On Mon, 2010-05-17 at 10:50 +0200, Stefan Vigerske wrote:
>>>>>> Hi,
>>>>>>>
>>>>> OK I hope that I understand that you would like to make things more
>>>>> simpler than remembering all different -I flags that are necessary
>>>> now.
>>>>
>>>> Yes.
>>>>
>>>>> So I have another suggestion - all projects can store their headers
>>>> that
>>>>> are going to be installed in a separate directory. 
>>>>> It would make things very simple from more point of views:
>>>>> - It would be immediately visible what files are going to be
>>>> installed
>>>>> - All subprojects would have to pass only one -I flag
>>>>>
>>>>> So it could look like this
>>>>>
>>>>> coin-cbc/
>>>>> ... 
>>>>> public_include_two.h
>>>>> some_clp_source.cpp
>>>>> some_clp_header.hpp
>>>>> ...
>>>>>
>>>>> The Makefile.am could contain something like
>>>>> AM_CPPFLAGS += -I includes
>>>>>
>>>>> and the source files would contain
>>>>> #include "coin/public_include_two.h"
>>>>>
>>>>> And the Cbc project's Makefile.am would contain
>>>>> AM_CPPFLAGS += -I $(CLPOBJDIR)/src/includes
>>>>
>>>> But I don't think that one can convince all (CoinAll related) project
>>>> managers to change their directory structure, I mean to tell them
>>>> where
>>>> they have to store header files.
>>>> I could more imagine that a project stores their include paths
>>>> somewhere
>>>> where another projects configure can find it. Currently we use the
>>>> xxx-uninstalled.pc files for this.
>>>> Then the fallback for the case that pkg-config is not there parses the
>>>> xxx-uninstalled.pc for the compiler flags.
>>>> Problem are hidden dependencies here again. If XXX depends on YYY, and
>>>> YYY depends on ZZZ, then XXX may find the flags to use YYY from the
>>>> yyy-uninstalled.pc file, but it may also require the compiler flags
>>>> from
>>>> ZZZ. That makes the configure.ac files a bit messy for the fallback
>>>> currently.
>>>
>>> I think that nobody will be able to resist to a smarter solution :-)
>>> Well, I have some other ideas how the directory tree could be
>>> reorganized, so I will post them with the appropriate reasoning and
>>> maybe project managers will like that.
>>> On the other hand, SVN does not handle file moves very well, so there
>>> may be a problem even if the reorganization idea is perfect.
>>> The other possibility that I can think of would be using another
>>> solution than a .pc file.
>>> The benefit of pkg-config is that they provide a standard way of
>>> checking for CFLAGS and LDFLAGS. We need only CPPFLAGS and other
>>> dependencies and we need it only for our purposes, nobody cares whether
>>> we use pkg-config or not there.
>>> So I suggest using a file containing only CPPFLAGS and dependencies.
>>> Then there could be a script (or a m4 macro) that would go through all
>>> dependencies recursively and fetching CPPFLAGS from them.
>>> It could look like this (copy&paste to your shell, all will happen in a
>>> subdirectory):
>>> http://pastebin.com/raw.php?i=rgkBXCL0
>>> Of course the calls of m4_append within cppflags.m4 should be wrapped,
>>> but this should be OK as a demonstration.
>>>
>>>>>>> it should be OK. We can't use libtool here, but
>>>>>>> pkg-config would not help us anyway.
>>>>>> It does, I can just ask pkg-config about the compiler flags for a
>>>> package.
>>>>>
>>>>> I meant it like that if a package is not installed, it is
>>>> troublesome to
>>>>> locate its .pc file.
>>>>
>>>> Since we know the base directory of the subproject and we assume that
>>>> the xxx-uninstalled.pc file is in this directory, it's not so
>>>> difficult
>>>> to locate.
>>>
>>> Yes, you are right, although it means that you have to do things in a
>>> way that users are not used to.
>>> There's nothing fundamentally wrong with it, it just tends to make the
>>> build system more cryptic.
>>>
>>>>> Moreover a "standard" .pc file is usually processed by the configure
>>>>> script and relates to the state after installation.
>>>>> Having a hand written .pc file for the "subproject build" does not
>>>> sound
>>>>> bad, though.
>>>>> Am I right that this is that what you have meant?
>>>>
>>>> Yes, this is what we do. :-)
>>>> Btw, there is a kind of small documentation of the pkg-config setup at
>>>> https://projects.coin-or.org/CoinTLC/wiki/BuildSystemUsingPkgConfig
>>>
>>> I have not studied it line by line but, I will do that as soon as I have
>>> more time. 
>>> What comes to my mind when I see that is that it would be nice to stick
>>> to "standard" macros more than creating custom ones if it is possible.
>>> I know that my macro contribution doesn't really show that I am a
>>> supporter of this approach, but at least let's not forget advantages of
>>> simplicity :-)
>>>
>>>> I think that gnulib has some interesting features that are missing in
>>>> the pkg-config setup. The main hurdle from my point of view is that
>>>> the
>>>> .la files do not communicate header paths.
>>>> I agree that this may be ok for building against installed projects
>>>> where the directories of the headers can be deducted from the library
>>>> paths. But for building against an uninstalled project, something is
>>>> missing.
>>>> I don't know, maybe one should abandon the idea of building against
>>>> uninstalled projects, at least with gnulib? The long term goal is to
>>>> get
>>>> rid of this anyway, if I remember correctly.
>>>
>>> As I say, it would be nice to have it. Imagine that you have a distro
>>> that doesn't have any COIN stuff in their repos. And imagine the pain of
>>> a user trying to compile some of it and therefore having to deal with
>>> all of COIN dependencies... 
>>> I think that the macro that I have provide a good start.
>>>
>>>>
>>>> Stefan
>>>
>>> Sorry for my lack of flexibility, but I have been busy above average the
>>> past week, it should settle down a bit now.
>>> Regards,
>>> Matej
>>>
>>>
>>>
>>
> 
>