Integration -- was: Re: [Coin-standards] minutes / strategy

[Leonardo B. Lopes]

Let me split this reply into a couple parts. Here we can keep talking
design and strategy, and I'll send another one where we can start talking
specifically about CLP.

On Wed, 10 Apr 2002, Irv Lustig wrote:

> >  But I wonder if they complain about some related things: Do you
> >hear complaints about users who wished they could do some Oracle
> >integration?
> 
> Yes.  And we provide solutions for that.
> 
> >  Or who could really benefit from running SPSS or Mathematica
> >over some data before using it in an optimization code?
> 
> Yes, and our customers do it all the time.
> 
> >  Or who would like
> >to develop web interfaces to their applications?
> 
> Yes, and our customers do it all the time.
> 

Two points here: 1st, the distinction between "we provide solutions for
that", and "our customers do it all the time", which IMHO is a critical
point. We've got to be able to do more of the former, and rely less on the
latter, especially when what the customer is doing is technical analysis.
And I don't think we can do it unless we update the way programs talk
optimization to each other. 

<snip>Irv Lustig argued that instances have much greater data requirements
than do models and provided an example</snip>

Point taken. Most of the time, it is easier to provide linkage to models
than to data. But there are cases in which you have to manipulate the
instance _despite_ having access to a modeling language or library. I'm
working on a case like that right now. This situation arises more and
more as you try to provide the user more abstraction. More on this
later...

> I don't see how a new standard makes it *easier* to manipulate optimization 
> problems.  If there is a new standard, I have to understand it in order to 
> use it.  And if the standard represents instances and not models, creating 
> instances using a new standard is just as hard as it is to use any of the 
> solver libraries.  This is why modeling languages are so appealing.  They 
> make the manipulation of optimization problems much easier.

The standard can be easier to manipulate if we design it with that goal in
mind. MPS was designed to be read using card readers or tape. Simply
assuming random access would make a tremendous difference. It allows
people to manage ultra-large instances, do decompositions, convey problem
structure, etc... Having it be xml based also helps with network
communication, with data extraction from other programs, with data
exporting into other programs, etc...

I don't expect to be creating instances bypassing a modeling language or
library very often, no matter what we have for a standard. But I do expect
to be collaborating with a modeling language or library in order to better
insulate the modeler of some of the things going on. Modeling languages
can't and shouldn't do everything. More on that later...

> Those standards will make it easier to integrate external data into 
> optimization tools.  And we need that.  This is one of the big problems our 
> customers face.  But I don't see how a standard helps integrate 
> optimization into non-optimization applications.  That is being addressed 
> by ILOG, Dash, and Maximal via products like OPL Studio, Xpress-Mosel, and 
> Optimax, respectively.  It is those tools that need to improve in order to 
> address the integration problem.

Making it easier to integrate external data into optimization models is
sufficient motivation for me. I think making it easier to integrate
external data implies that it is easier to integrate optimization into
non-optimization applications. I also think that the more people do that,
the more people will be using OPL-Studio, Xpress-Mosel and Optimax. The
people who develop and sell those tools are those who will most benefit
from what we are trying to do. If academia benefits the most, that is fine
with me too. I think everybody benefits.

By the way, I haven't had the opportunity to thank you (Irv) for your
insights. I've been learning a lot and changing a bit my assumptions about
some of these issues. I hope you're not done, although I'm anxious to get
technical soon. I'll try to do that next...

Cheers,
Leo.

========================================================================
Leonardo B. Lopes                                       leo at iems.nwu.edu 
Ph.D. Student                                              (847)491-8470
IEMS - Northwestern University              http://www.iems.nwu.edu/~leo