Bayesian model reduction

Bayesian model reduction is a method for computing the evidence and posterior over the parameters of Bayesian models that differ in their priors.

[1][2] A full model is fitted to data using standard approaches.

Hypotheses are then tested by defining one or more 'reduced' models with alternative (and usually more restrictive) priors, which usually – in the limit – switch off certain parameters.

If the priors and posteriors are normally distributed, then there is an analytic solution which can be computed rapidly.

This has multiple scientific and engineering applications: these include scoring the evidence for large numbers of models very quickly and facilitating the estimation of hierarchical models (Parametric Empirical Bayes).

Therefore, the posteriors are estimated using approaches such as MCMC sampling or variational Bayes.

A reduced model can then be defined with an alternative set of priors

: The objective of Bayesian model reduction is to compute the posterior

can be expressed as the product of the full posterior, the ratio of priors and the ratio of evidences: The evidence for the reduced model is obtained by integrating over the parameters of each side of the equation: And by re-arrangement: Under Gaussian prior and posterior densities, as are used in the context of variational Bayes, Bayesian model reduction has a simple analytical solution.

[1] First define normal densities for the priors and posteriors: where the tilde symbol (~) indicates quantities relating to the reduced model and subscript zero – such as

For convenience we also define precision matrices, which are the inverse of each covariance matrix: The free energy of the full model

is an approximation (lower bound) on the log model evidence:

that is optimised explicitly in variational Bayes (or can be recovered from sampling approximations).

, which is the Normal distribution with mean zero and standard deviation 0.5 (illustrated in the Figure, left).

The model with this prior is fitted to the data, to provide an estimate of the parameter

The hypothesis that the parameter contributed to the model is then tested by comparing the full and reduced models via the Bayes factor, which is the ratio of model evidences: The larger this ratio, the greater the evidence for the full model, which included the parameter as a free parameter.

Conversely, the stronger the evidence for the reduced model, the more confident we can be that the parameter did not contribute.

Note this method is not specific to comparing 'switched on' or 'switched off' parameters, and any intermediate setting of the priors could also be evaluated.

[4] The experimenter specifies multiple competing models which differ in their priors – e.g. in the choice of parameters which are fixed at their prior expectation of zero.

Having fitted a single 'full' model with all parameters of interest informed by the data, Bayesian model reduction enables the evidence and parameters for competing models to be rapidly computed, in order to test hypotheses.

These models can be specified manually by the experimenter, or searched over automatically, in order to 'prune' any redundant parameters which do not contribute to the evidence.

Bayesian model reduction has been used to explain functions of the brain.

By analogy to its use in eliminating redundant parameters from models of experimental data, it has been proposed that the brain eliminates redundant parameters of internal models of the world while offline (e.g. during sleep).

[5][6] Bayesian model reduction is implemented in the Statistical Parametric Mapping toolbox, in the Matlab function spm_log_evidence_reduce.m .

Example priors. In a 'full' model, left, a parameter has a Gaussian prior with mean 0 and standard deviation 0.5. In a 'reduced' model, right, the same parameter has prior mean zero and standard deviation 1/1000. Bayesian model reduction enables the evidence and parameter(s) of the reduced model to be derived from the evidence and parameter(s) of the full model.