Re: [Neurostat-develop] first ideas...

[Fabrice Rossi]

Joseph wrote:
> > 1) With dense matrices, we can use efficient BLAS implementations (ATLAS 
> > seems
> > to give quite good results compared to vendor optimized BLAS). There is a
> > sparseblas on netlib (in fortran), but I don't think it is close to the
> > optimization level of ATLAS (authors of ATLAS plan to work on sparse 
> > matrices,
> > but it is not the case right now, I think).
> 
> generally, the size of the matrices is not really huge (100x100 is the
> maximum), so the use of optimized library is not so critic. In my
> experience with my C++ program interfaced with BLAS, if an optimization
> task takes 5 minutes with the Intel optimized BLAS routine, it takes
> about 8 minutes with the netlib BLAS and 6 minutes with ATLAS. So, the
> gain of speed is just between 30% to 40%, it's not a big deal.
> MLP are not used for real time task and it doesn't really matter to wait
> 8 minutes instead of 5 minutes.

I agree that the speed up is not so important, but still for massive search
such as genetic optimization of the network architecture, even 20% speed gain
is important. Nevertheless, I don't think there is problem to work both with a
dense storage and with a spare one, mainly thanks to "object" approach.

> I don't think that we need to use the very general sparse netlib
> library, we can implement only three or four BLAS-operations one a
> particular sparse-architecture (see bellow).

You're right. 
 
> > 3) In general, if we want to use optimization algorithms, we need to rely 
> > on a
> > flat representation of the numerical parameters of the MLP (a "vector"). I
> > don't know many things about sparse matrices and so I wonder how this
> > requierement can be achieved.
> >
> 
> In fact the problem evocated is the main reason to use sparse matrices,
> because the representation of the coefficients in a sparse matrix is
> flat!!
> 
> Indeed, we have essentially the choice between the compressed column
> storage or compressed row storage (see  SparseLib++ v.1.5. Sparse Matrix
> Class Library, Reference Guide, / R. Pozo, K.A. Remington  April 1996,
> http://math.nist.gov/sparselib++/).
> 
> With these representations,  the parameters are inevitably stored in a
> "flat" vector but are also  "compressed" (without zero).

You're perfectly right and so there is basically no problem to support both
dense and sparse matrices and vectors.

> By the way you can represent the weights of the MLP in a flat vector,
> without zeros, and construct the sparse matrices and vectors
> representing the architecture of the MLP using  (double *) pointers on
> the (good) coordinates of this vector.

That's exactly what we need.

Ok, so here is how I see a possible header:

int propagate(MLP *m,double *param,int param_size,double *input,int
in_size,double *result,int out_size)

The return value is an error code or something similar (maybe it is not
needed). (double *,int) pairs are "vectors". Everything else is embedded in
the MLP struct. It contains of course a description of the architecture, which
is basically a way to interpret the parameter vector (i.e. param), which can
contain dense matrices and vectors, or sparse ones. We also embed in the
struct all the needed cache (pre-output of each layer, etc.). 

I don't know if we really need to have all the sizes because they are already
in the MLP struct. But it allows at least to test for adequacy. Maybe it is
not needed in the low level API.

Comments?

Fabrice