[Cbc] Setting Up Large Scale Problems

[Matt Bromberg]

I have a fairly large dense linear programming problem, of continuous 
variables.
My constraint matrix is about 10K x 10K double precision  and is saved 
as a Matlab file, or equivalently I can get it into Python as an ndarray.

My really dumb question is how does one usually get a problem of this 
size into a solver like cbc/clp or even into one of the commercial solvers?

I tried to save it in one of the text formats,  MPS,  and it wanted to 
consume close to 200Gbytes of hard drive space, when in fact the 
constraint matrix itself is only 0.8 Gbytes of binary data.

Also of interest is the fact that internally the two solvers I've 
tried,  Matlabs linprog() and glpk() quickly consume more than 64 Gbytes 
of RAM trying to solve this,  prior to dying.  Thus they are horribly 
memory inefficient IMHO.  I keep hearing that commercial solvers can 
deal with millions of variables and constraints.  Their problems must 
all be very sparse,  mine is quite dense.

I am new to these linear solvers since I've always rolled my own 
optimizers for special case nonlinear problems.  I suppose I should also 
ask whether the simplex method is really the best for problems of this 
size, and whether I shouldn't be using some kind of interior point method.

Thanks in advance.