In machine learning, ensemble averaging is the process of creating multiple models (typically artificial neural networks) and combining them to produce a desired output, as opposed to creating just one model.
[citation needed] Ensemble averaging is one of the simplest types of committee machines.
[2] The theory of ensemble averaging relies on two properties of artificial neural networks:[3] This is known as the bias–variance tradeoff.
[4] The idea of combining experts can be traced back to Pierre-Simon Laplace.
[5] The theory mentioned above gives an obvious strategy: create a set of experts with low bias and high variance, and average them.
Generally, what this means is to create a set of experts with varying parameters; frequently, these are the initial synaptic weights of a neural network, although other factors (such as learning rate, momentum, etc.)
Some authors recommend against varying weight decay and early stopping.
[3] The steps are therefore: Alternatively, domain knowledge may be used to generate several classes of experts.
[2] A more recent ensemble averaging method is negative correlation learning,[6] proposed by Y. Liu and X. Yao.