Wake-sleep algorithm

[6] The goal of the wake-sleep algorithm is to find a hierarchical representation of observed data.

[3] Neurons are fired by recognition connections (from what would be input to what would be output).

Generative connections (leading from outputs to inputs) are then modified to increase probability that they would recreate the correct activity in the layer below – closer to actual data from sensory input.

[1] The process is reversed in the “sleep” phase – neurons are fired by generative connections while recognition connections are being modified to increase probability that they would recreate the correct activity in the layer above – further to actual data from sensory input.

[1] Since the recognition network is limited in its flexibility, it might not be able to approximate the posterior distribution of latent variables well.