[Gneuralnetwork] Bayesian models and Monte Carlo methods

[Tobias Wessels]

Hi everyone,

First a short remark on my previous question and then, more
importantly, an issue which I have with the current implementation of
neural networks.

I have read more about Monte Carlo methods and my previous question is
answered by the (already cited) book of McKay. My misconception was
that
        $\sum_{i=1}^L \tilde q(w_i)$
would approximate  the total weight of $\tilde q$, which it doesn't,
because the $w_i$ are already drawn according to the distribution of
$q$.


Now to the more important issue:
As it seems to me, Monte Carlo methods are mainly used for models,
which use Bayesian inference, as in these models you have to calculate
probabilistic integrals frequently, which is a difficult task unless
the model is broken down to a very narrow class of distribution
functions (i.e. Gaussian). Monte Carlo methods are a tool to
approximate these integrals by simple (that is: computationally
inexpensive) means.

The issue that I have now with the current implementation of neural
networks is that it is not adapted to Bayesian inference models. In
these models, the networks are not trained to use a single, most likely
choice of weight vector as in the maximum likelihood method, but
instead these models consider a probability distribution for the
weights of the neurons and then calculates an average (expected value)
of the output, given a new input x. (This is the important step, where
Monte Carlo methods are used, because this expectation is basically an
integral over a very complex probability density). 

In the current implementation, however, the nodes have a single
parameter/array, which is set to the most likely choice, rather than
using a distribution of possible values for the weights. I don't see
how I would implement a Bayesian inference model using the current
code.

I hope that my explanations were somewhat clear and somewhat accurate,
as I am still new to this topic myself. So if you see it differently,
please help me understanding this topic.

Kind regards,

Tobias