The Signal and the Noise: Why So Many Predictions Fail – but Some Don't is a 2012 book by Nate Silver detailing the art of using probability and statistics as applied to real-world circumstances.

The book includes case studies from baseball, elections, climate change, the 2008 financial crash, poker, and weather forecasting.

The book includes richly detailed case studies from baseball, elections, climate change, the financial crash, poker, and weather forecasting.

There is much on the need for improved expressions of uncertainty in all statistical statements, reflecting ranges of probable outcomes and not just single "point estimates" like averages.

The problem Silver finds is a belief in perfect experimental, survey, or other designs, when data often comes from a variety of sources and idealized modeling assumptions rarely hold true.

Often such models reduce complex questions to overly simple "hypothesis tests" using arbitrary "significance levels" to "accept or reject" a single parameter value.

In contrast, the practical statistician first needs a sound understanding of how baseball, poker, elections or other uncertain processes work, what measures are reliable and which not, what scales of aggregation are useful, and then to utilize the statistical tool kit as well as possible.

This Bayesian approach is named for the 18th-century minister Thomas Bayes who is credited for first deriving a simple formula for updating probabilities using new data.

For Silver, the well-known method needs revitalizing as a broader paradigm for thinking about uncertainty, founded on learning and understanding gained incrementally, rather than through any single set of observations or an ideal model summarized by just a few key parameters.

Part of that learning is the informal process of changing assumptions or the modeling approach, in the spirit of a craft whose goal is to devise the best betting odds on well-defined future events and their outcomes.

In praise of the book, Murray Cantor, IBM Distinguished Engineer, wrote Nate Silver's The Signal and Noise is an excellent description of how prediction works.

In 2012, after his triumph of predicting the outcome of the last two presidential elections and selling his "fivethirtyeight" blog to the New York Times, Nate Silver accomplished what is almost impossible.