An MEMM is a discriminative model that extends a standard maximum entropy classifier by assuming that the unknown values to be learnt are connected in a Markov chain rather than being conditionally independent of each other.
MEMMs find applications in natural language processing, specifically in part-of-speech tagging[1] and information extraction.
is a normalization term ensuring that the distribution sums to one.
This form for the distribution corresponds to the maximum entropy probability distribution satisfying the constraint that the empirical expectation for the feature is equal to the expectation given the model: The parameters
[4] Furthermore, a variant of the Baum–Welch algorithm, which is used for training HMMs, can be used to estimate parameters when training data has incomplete or missing labels.
can be found using a very similar Viterbi algorithm to the one used for HMMs.
The dynamic program uses the forward probability: An advantage of MEMMs rather than HMMs for sequence tagging is that they offer increased freedom in choosing features to represent observations.
In sequence tagging situations, it is useful to use domain knowledge to design special-purpose features.
In the original paper introducing MEMMs, the authors write that "when trying to extract previously unseen company names from a newswire article, the identity of a word alone is not very predictive; however, knowing that the word is capitalized, that is a noun, that it is used in an appositive, and that it appears near the top of the article would all be quite predictive (in conjunction with the context provided by the state-transition structure).
[2] Therefore, MEMMs allow the user to specify many correlated, but informative features.
Another advantage of MEMMs versus HMMs and conditional random fields (CRFs) is that training can be considerably more efficient.
A drawback of MEMMs is that they potentially suffer from the "label bias problem," where states with low-entropy transition distributions "effectively ignore their observations."
Conditional random fields were designed to overcome this weakness,[5] which had already been recognised in the context of neural network-based Markov models in the early 1990s.
[5][6] Another source of label bias is that training is always done with respect to known previous tags, so the model struggles at test time when there is uncertainty in the previous tag.