Independent and identically distributed random variables

In probability theory and statistics, a collection of random variables is independent and identically distributed (i.i.d., iid, or IID) if each random variable has the same probability distribution as the others and all are mutually independent.

[1] IID was first defined in statistics and finds application in many fields, such as data mining and signal processing.

Statistics commonly deals with random samples.

More formally, it is "a sequence of independent, identically distributed (IID) random data points."

In other words, the terms random sample and IID are synonymous.

In statistics, "random sample" is the typical terminology, but in probability, it is more common to say "IID."

Independent and identically distributed random variables are often used as an assumption, which tends to simplify the underlying mathematics.

In practical applications of statistical modeling, however, this assumption may or may not be realistic.

assumption is also used in the central limit theorem, which states that the probability distribution of the sum (or average) of i.i.d.

variables with finite variance approaches a normal distribution.

assumption frequently arises in the context of sequences of random variables.

[5] For example, repeated throws of loaded dice will produce a sequence that is i.i.d., despite the outcomes being biased.

– The signal level must be balanced on the time axis.

, respectively, and denote their joint cumulative distribution function by

, see also Independence (probability theory) § Two random variables.)

The definition extends naturally to more than two random variables.

denotes the joint cumulative distribution function of

A sequence of outcomes of spins of a fair or unfair roulette wheel is i.i.d.

Toss a coin 10 times and write down the results into variables

Roll a die 10 times and save the results into variables

Many results that were first proven under the assumption that the random variables are i.i.d.

have been shown to be true even under a weaker distributional assumption.

The most general notion which shares the main properties of i.i.d.

[citation needed] Exchangeability means that while variables may not be independent, future ones behave like past ones — formally, any value of a finite sequence is as likely as any permutation of those values — the joint probability distribution is invariant under the symmetric group.

This provides a useful generalization — for example, sampling without replacement is not independent, but is exchangeable.

To train ML models effectively, it is crucial to use data that is broadly generalizable.

hypothesis allows for a significant reduction in the number of individual cases required in the training sample, simplifying optimization calculations.

In optimization problems, the assumption of independent and identical distribution simplifies the calculation of the likelihood function.

Due to this assumption, the likelihood function can be expressed as:

There are two main reasons why this hypothesis is practically useful with the central limit theorem (CLT):