The noisy channel model is a framework used in spell checkers, question answering, speech recognition, and machine translation.
In this model, the goal is to find the intended word given a word where the letters have been scrambled in some manner.
of valid words be some subset of
is the scrambled word that was actually received.
The goal of the noisy channel model is to find the intended word given the scrambled word that was received.
Methods of constructing a decision function include the maximum likelihood rule, the maximum a posteriori rule, and the minimum distance rule.
In some cases, it may be better to accept the scrambled word as the intended word rather than attempt to find an intended word in the dictionary.
makes up the dictionary of valid English words.
There are several mistakes that may occur while typing, including: To construct the noisy channel matrix
, we must consider the probability of each mistake, given the intended word (
These probabilities may be gathered, for example, by considering the Damerau–Levenshtein distance between
or by comparing the draft of an essay with one that has been manually edited for spelling.
One naturally wonders if the problem of translation could conceivably be treated as a problem in cryptography.
When I look at an article in Russian, I say: 'This is really written in English, but it has been coded in some strange symbols.
[2] Suppose we want to translate a foreign language to English, we could model
However, by Bayes law, we have the equivalent equation:
The benefit of the noisy-channel model is in terms of data: If collecting a parallel corpus is costly, then we would have only a small parallel corpus, so we can only train a moderately good English-to-foreign translation model, and a moderately good foreign-to-English translation model.
However, we can collect a large corpus in the foreign language only, and a large corpus in the English language only, to train two good language models.
Combining these four models, we immediately get a good English-to-foreign translator and a good foreign-to-English translator.
[3] The cost of noisy-channel model is that using Bayesian inference is more costly than using a translation model directly.
, it would have to read out predictions by both the translation model and the language model, multiply them, and search for the highest number.
Speech recognition can be thought of as translating from a sound-language to a text-language.
is the probability that a speech sound S is produced if the speaker is intending to say text T. Intuitively, this equation states that the most likely text is a text that's both a likely text in the language, and produces the speech sound with high probability.
The utility of the noisy-channel model is not in capacity.
Theoretically, any noisy-channel model can be replicated by a direct
However, the noisy-channel model factors the model into two parts which are appropriate for the situation, and consequently it is generally more well-behaved.
When a human speaks, it does not produce the sound directly, but first produces the text it wants to speak in the language centers of the brain, then the text is translated into sound by the motor cortex, vocal cords, and other parts of the body.
This is justified in the practical success of noisy-channel model in speech recognition.
Consider the sound-language sentence (written in IPA for English) S = aɪ wʊd laɪk wʌn tuː.
With a good English language model, we would have