[Coin-standards] minutes / strategy

[Irv Lustig]

Leo:

At 12:35 AM 4/10/02 -0500, Leonardo B. Lopes wrote:
>Great! At least one person read that long message :)... It'd be nice to
>get other people's opinions on some of these things...

I agree!


>I didn't necessarily expect them to do that very often, especially to you.
>I probably wouldn't complain about that issue to you if I were an user
>either.

I talk to lots of optimization users, and I do hear the issues that bother 
them.  People do express their "pains" in terms of using optimization software.

>  But I wonder if they complain about some related things: Do you
>hear complaints about users who wished they could do some Oracle
>integration?

Yes.  And we provide solutions for that.

>  Or who could really benefit from running SPSS or Mathematica
>over some data before using it in an optimization code?

Yes, and our customers do it all the time.

>  Or who would like
>to develop web interfaces to their applications?

Yes, and our customers do it all the time.

>  Do you think you might be
>able to sell more cplex licenses if people didn't have to install it on
>their own machines?

NO!  There is a real challenge to make sure that the growth of server-based 
optimization does not erode into our revenues that are traditionally based 
on client-based deployments.  The only thing that prevents having CPLEX on 
a web server is our license agreement, not technology.


>I think that _part_ of creating more satisfactory answers to those
>questions, and thus creating new growth opportunities and new research
>material for optimization, is coming up with better ways to ship
>optimization instances around.

Only if the cost of shipping them around between applications is cheap.

What seems to be happening is that people have data in databases that needs 
to be transformed in some way to an optimization instance, and then the 
optimization instance needs to be solved.  The data that drives the 
instance is much smaller than the description of the instance.  So, what we 
see people doing is setting up servers where the server queries the 
(possibly remote) database to get the relevant data, builds the instance, 
and then solves the problem.

Here's a simple example that I use all the time from AMPL.  Take a look at 
the model steelT.mod and datafile steelT.dat.  Here, you have 31 data items 
needed to describe a problem that has 12 constraints, 24 variables, 38 
matrix nonzeros, 24 objective nonzeros, 5 RHS nonzeros, 8 bound nonzeros. 
So, now the number of nonzero values needed to represent the problem is 75. 
(And I'm not even counting the storage required to tell where all these 
nonzeros are located) The requirements for describing the problem have more 
than doubled from the data instance.  Now, scale up the problem to 100 
products and 500 time periods.  The data requirements increase, but the 
problem instance requirements increase even more.  If someone were to 
create a server for solving this problem, it would be smarter to set up the 
server to receive the equivalent (in XML, say) of "steelT.dat", rather than 
the equivalent of the complete problem instance.  Or, even better, have the 
data live on the same server as the optimization solver.

>  That is why IMHO, amongst other
>developments we need a better, open, comprehensive standard.

I still think the standard is a solution looking for an industrial 
problem.  However, IMHO, the standard *does* help academic research.


>That is not really my vision for the standard. I think of communicating
>instances, not models. That distinction gets complicated once programs get
>sufficiently sofisticated (especially with CLP), and that is one of the
>things we need to figure out.

I agree.


>That's important. I hadn't though about it that way. I hope you (Irv) can
>raise a flag for that once we move to technical aspects. Do you think
>block structures would make users uncomfortable?

Maybe. It's hard to say because it's not clear how well people can describe 
block structures.


>Forgiven. I share this concern, in the cases where files would actually be
>written and read. But I believe we can engineer to keep alpha pretty low.
>BTW, how important is read/write time in the business environment?

Today, especially when people use modeling languages, we often see more 
time spent in model generation than in the solver itself.  Read/write time 
just adds to that generation time.  It needs to be kept low.


>I don't think of the format as any more useful for modeling than MPS is. I
>would hope people don't think about the block structure when they are
>modeling. But the structure can be useful for computers to look at, and is
>something that can be extracted from the model.

I'm not sure how well we know how to do that yet, but I admit I'm not up on 
the literature in this regard.
>
>It does, for a lot of cases. Just like ILOG's or IBM's libraries do for
>some other cases. But they all solve a different problem than this
>standard would. My vision of a standard is for machine <-> machine
>communication. The standard would be used by people developing tools that
>complement or augment ampl or ILOG or IBM software. I think there is a lot
>of value to be added in that realm, but it isn't going to happen if
>manipulating optimization problems is as hard as it is now.

I don't see how a new standard makes it *easier* to manipulate optimization 
problems.  If there is a new standard, I have to understand it in order to 
use it.  And if the standard represents instances and not models, creating 
instances using a new standard is just as hard as it is to use any of the 
solver libraries.  This is why modeling languages are so appealing.  They 
make the manipulation of optimization problems much easier.


>I agree that people being able to apply optimization is the number one
>limitation we currently have, and I hope we are both right, since I intend
>to expend quite a bit of intellectual effort over the next couple decades
>working on that problem. I think that a big part of solving this problem
>is creating domain-specific extensions to current modeling tools.

I think this is a nice idea.

>  To do
>that, we will often need to interface with non-optimization applications.
>It better be cheap and easy to do that, and good standards are part of the
>solution.

Those standards will make it easier to integrate external data into 
optimization tools.  And we need that.  This is one of the big problems our 
customers face.  But I don't see how a standard helps integrate 
optimization into non-optimization applications.  That is being addressed 
by ILOG, Dash, and Maximal via products like OPL Studio, Xpress-Mosel, and 
Optimax, respectively.  It is those tools that need to improve in order to 
address the integration problem.


>I also think that the applications are there *today*, and that we need a
>lot more than a standard to make them successful. A standard doesn't solve
>all the problems, but it helps create an environment where the problems
>can be solved. Without it, there are a lot of things that would be just
>too cumbersome to do. A standard is not a sufficient, but a necessary
>condition.

I disagree that a standard for representing problem instances is necessary 
for the growth of optimization applications.  We have lots of tools and 
environments (from ILOG as well as ILOG's competitors) that make 
optimization easier to use and integrate into applications.  IMHO, if we 
could agree on a standard modeling language where each vendor could provide 
their own implementation of that language, we'd be far better off.  But I 
don't see how all the vendors could agree on what that common language 
should look like.  It needs to have features that are present in AMPL, OPL, 
MPL, and Mosel.  And someone needs to put a reference implementation of 
such a language in the public domain, or at least create a scheme for 
licensing that implementation so that the vendors could modify it.  This is 
what Sun has done with Java.  AMPL almost has these properties, but the 
implementation can't be modified by the vendors, and there are some design 
deficiencies that should be fixed.

         -Irv