Bayesian programming is a formalism and a methodology for having a technique to specify probabilistic models and solve problems when less than the necessary information is available.
Edwin T. Jaynes proposed that probability could be considered as an alternative and an extension of logic for rational reasoning with incomplete and uncertain information.
In his founding book Probability Theory: The Logic of Science[1] he developed this theory and proposed what he called “the robot,” which was not a physical device, but an inference engine to automate probabilistic reasoning—a kind of Prolog for probability instead of logic.
The constituent elements of a Bayesian program are presented below:[4] The purpose of a description is to specify an effective method of computing a joint probability distribution on a set of variables
Recursive application of Bayes' theorem leads to: Conditional independence hypotheses then allow further simplifications.
as the conjunction of these chosen variables and setting: We then obtain: Such a simplification of the joint distribution as a product of simpler distributions is called a decomposition, derived using the chain rule.
Learning takes place when some of these parameters are computed using the data set
This is similar to calling a subroutine in classical programming and provides an easy way to build hierarchical models.
, it is always possible to compute any possible question using the following general inference: where the first equality results from the marginalization rule, the second results from Bayes' theorem and the third corresponds to a second application of marginalization.
Replacing the joint distribution by its decomposition we get: which is usually a much simpler expression to compute, as the dimensionality of the problem is considerably reduced by the decomposition into a product of lower dimension distributions.
The purpose of Bayesian spam filtering is to eliminate junk e-mails.
Starting from an initial standard setting, the classifier should modify its internal parameters when the user disagrees with its own decision.
Starting from the joint distribution and applying recursively Bayes' theorem we obtain: This is an exact mathematical expression.
However, the programmer may assume this hypothesis and may develop the model and the associated inferences to test how reliable and efficient it is.
Both methods could be combined: the system could start with initial standard values of these parameters issued from a generic database, then some incremental learning customizes the classifier to each individual user.
They are defined by the following Bayesian program: With these hypotheses and by using the recursive formula, it is possible to solve the inference problem analytically to answer the usual
This leads to an extremely efficient algorithm, which explains the popularity of Kalman filters and the number of their everyday applications.
Hidden Markov models (HMMs) are another very popular specialization of Bayesian filters.
Since 2000, Bayesian programming has been used to develop both robotics applications and life sciences models.
[24] In life sciences, bayesian programming was used in vision to reconstruct shape from motion,[25] to model visuo-vestibular interaction[26] and to study saccadic eye movements;[27] in speech perception and control to study early speech acquisition[28] and the emergence of articulatory-acoustic systems;[29] and to model handwriting perception and control.
It employs the principles of compositionality (building abstract representations from parts), causality (building complexity from parts) and learning to learn (using previously recognized concepts to ease the creation of new concepts).
[31] The comparison between probabilistic approaches (not only bayesian programming) and possibility theories continues to be debated.
They argue that probability is insufficient or inconvenient to model certain aspects of incomplete/uncertain knowledge.
The defense of probability is mainly based on Cox's theorem, which starts from four postulates concerning rational reasoning in the presence of uncertainty.
It demonstrates that the only mathematical framework that satisfies these postulates is probability theory.
Extended classical programming languages include logical languages as proposed in Probabilistic Horn Abduction,[35] Independent Choice Logic,[36] PRISM,[37] and ProbLog which proposes an extension of Prolog.
It can also be extensions of functional programming languages (essentially Lisp and Scheme) such as IBAL or CHURCH.
The underlying programming languages can be object-oriented as in BLOG and FACTORIE or more standard ones as in CES and FIGARO.
Jaynes' precept of "probability as logic" argues that probability is an extension of and an alternative to logic above which a complete theory of rationality, computation and programming can be rebuilt.
The precise comparison between the semantics and power of expression of Bayesian and probabilistic programming is an open question.