Re: [SwarmFest2004] My submission (Can swarm based systems outperform other methods i n training neural networks?)

[Rick Riolo]

hi danil,

ok, we have your submission for swarmfest,

thanks!
 - r

Rick Riolo                           address@hidden
Center for the Study of Complex Systems (CSCS)
4477 Randall Lab
University of Michigan         Ann Arbor MI 48109-1120
Phone: 734 763 3323                  Fax: 734 763 9267
http://cscs.umich.edu/~rlr

On Mon, 29 Mar 2004, Prokhorov, Danil (D.V.) wrote:

> Date: Mon, 29 Mar 2004 12:12:00 -0500
> From: "Prokhorov, Danil (D.V.)" <address@hidden>
> To: "'address@hidden'" <address@hidden>
> Subject: [SwarmFest2004] My submission (Can swarm based systems
>     outperform other methods i n training neural networks?)
>
> Type: Research or Application
> Format: Abstract (Poster is fine too).
>
> Title: Can swarm based systems outperform other methods in training neural 
> networks?
>
> Danil V. Prokhorov
> Research and Advanced Engineering
> Ford Motor Company
> 2101 Village Rd., MD 2036
> Dearborn, MI 48124
> address@hidden
>
> Particle swarm optimization (PSO) has been applied to a wide variety of
> problems since its inception in 1995 [1].  Yet, it seems to be a deficit
> of applications of PSO to neural network training problems, especially
> in cases of medium- and large-size networks (more than 1000 weights),
> large training data sets (more than 100,000 data vectors) and recurrent
> neural networks.
>
> We are interested in efficient training methods for neural networks,
> especially those methods that scale well to problems requiring large
> data sets and recurrent neural networks.  We have developed the training
> methods based on the extended Kalman filter (EKF) algorithm and applied
> them successfully to many problems in system modeling and control using
> neural networks [2]-[4].  The EKF methods operate fundamentally in the
> pattern-by-pattern mode of data presentation (as opposed to the PSO
> which operates in the batch mode), although presenting training data in
> mini-batches (streams) has been found to be very effective [2].  The EKF
> training complexity scales roughly as O(square of number of weights).
>
> Recently, there have been claims of superior behavior of PSO applied to
> simple neural network training problems (see, e.g., [5]).  On the
> contrary, our own research demonstrates that, while the PSO may be
> effective in comparison with simple gradient based algorithms like the
> standard gradient descent and other first-order techniques, it is
> substantially inferior to the EKF and, possibly, other more advanced
> methods, especially when dealing with complex problems like ones
> discussed in [3].  Having much more experience with the KF based
> techniques than with the PSO, we might well be unaware of the right set
> of tricks swarm researchers employ to deal with large-scale optimization
> problems.  However, it is also possible that that, at least partially,
> the reason behind the observed PSO disadvantage lies in its batch mode
> of operation and poorly understood initialization of particles for large
> optimization problems.
>
> We wish to discuss our comparative results with those presented in [5]
> and offer to future PSO benchmark studies a couple of challenging
> problems for training recurrent neural networks already efficiently
> solved by the EKF.
>
>
> [1]   J. Kennedy, RC Eberhart, and Y. Shi. Swarm Intelligence. San Francisco, 
> Morgan
> Kaufmann, 2001.
>
> [2] Feldkamp and Puskorius, "A Signal Processing Framework Based on
> Dynamic Neural Networks with Application to Problems in Adaptation,
> Filtering and Classification," Proc. IEEE, Vol. 86, No. 11, pp.
> 2259-2277, 1998.
>
> [3] Prokhorov, D., Feldkamp, L., and I. Tyukin, "Adaptive Behavior with
> Fixed Weights in Recurrent Neural Networks: An Overview," Proc. of
> International Joint Conference on Neural Networks (IJCNN), WCCI'02,
> Honolulu, Hawaii, May 2002.
>
> [4] D. Prokhorov, G. Puskorius, and L. Feldkamp, "Dynamical Neural
> Networks for Control," in J. Kolen and S. Kremer (Eds.) A Field Guide to
> Dynamic Recurrent Networks, IEEE Press, 2001.
>
> [5] Gudise, V. G. and Venayagamoorthy, G. K. "Comparison of particle
> swarm optimization and backpropagation as training algorithms for neural
> networks." Proceedings of the IEEE Swarm Intelligence Symposium 2003
> (SIS 2003), Indianapolis, Indiana, USA. pp. 110-117, 2003.

> _______________________________________________ SwarmFest2004 mailing
> list address@hidden
> http://www.swarm.org/mailman/listinfo/swarmfest2004
>