EM algorithm and GMM model

In statistics, EM (expectation maximization) algorithm handles latent variables, while GMM is the Gaussian mixture model.

In the picture below, are shown the red blood cell hemoglobin concentration and the red blood cell volume data of two groups of people, the Anemia group and the Control Group (i.e. the group of people without Anemia).

As expected, people with Anemia have lower red blood cell volume and lower red blood cell hemoglobin concentration than those without Anemia.

red blood cell volume

red blood cell hemoglobin concentration

{\displaystyle x:={\big (}{\text{red blood cell volume}},{\text{red blood cell hemoglobin concentration}}{\big )}}

, and from medical studies[citation needed] it is known that

belongs to Anemia Group and

belongs to Control Group.

A maximum likelihood estimation can be applied: As the

are known, the log likelihood function can be simplified as below: Now the likelihood function can be maximized by making partial derivative over

is known, the estimation of the parameters results to be quite simple with maximum likelihood estimation.

a latent variable (i.e. not observed), with unlabeled scenario, the Expectation Maximization Algorithm is needed to estimate

Generally, this problem is set as a GMM since the data in each group is normally distributed.

[3][circular reference] In machine learning, the latent variable

is considered as a latent pattern lying under the data, which the observer is not able to see very directly.

ϕ , μ ,

are the parameter of the model.

With the EM algorithm, some underlying pattern

can be found, along with the estimation of the parameters.

The wide application of this circumstance in machine learning is what makes EM algorithm so important.

The EM algorithm consists of two steps: the E-step and the M-step.

Firstly, the model parameters and the

In the E-step, the algorithm tries to guess the value of

based on the parameters, while in the M-step, the algorithm updates the value of the model parameters based on the guess of

These two steps are repeated until convergence is reached.

The algorithm in GMM is: Repeat until convergence: [1] With Bayes Rule, the following result is obtained by the E-step:

; ϕ , μ ,

According to GMM setting, these following formulas are obtained:

In this way, a switch between the E-step and the M-step is possible, according to the randomly initialized parameters.