Spillover (experiment)
In ordinary settings where the researcher seeks to estimate the average treatment effect (
), violation of the non-interference assumption means that traditional estimators for the ATE, such as difference-in-means, may be biased.
However, there are many real-world instances where a unit's revelation of potential outcomes depend on another unit's treatment assignment, and analyzing these effects may be just as important as analyzing the direct effect of treatment.
One solution to this problem is to redefine the causal estimand of interest by redefining a subject's potential outcomes in terms of one's own treatment status and related subjects' treatment status.
Once the potential outcomes are redefined, the rest of the statistical analysis involves modeling the probabilities of being exposed to treatment given some schedule of treatment assignment, and using inverse probability weighting (IPW) to produce unbiased (or asymptotically unbiased) estimates of the estimand of interest.
Estimating spillover effects in experiments introduces three statistical issues that researchers must take into account.
Non-interference is violated when subjects can communicate with each other about their treatments, decisions, or experiences, thereby influencing each other's potential outcomes.
If the non-interference assumption does not hold, units no longer have just two potential outcomes (treated and control), but a variety of other potential outcomes that depend on other units’ treatment assignments,[9] which complicates the estimation of the average treatment effect.
Estimating spillover effects requires relaxing the non-interference assumption.
The researcher must posit a set of potential outcomes that limit the type of interference.
If the study population consists of all students living with a roommate in a college dormitory, one can imagine four sets of potential outcomes, depending on whether the student or their partner received the information (assume no spillover outside of each two-person room): Now an individual's outcomes are influenced by both whether they received the treatment and whether their roommate received the treatment.
Similarly, we can measure how ones’ outcomes change depending on their roommate's treatment status, when the individual themselves are treated.
While researchers typically embrace experiments because they require less demanding assumptions, spillovers can be “unlimited in extent and impossible to specify in form.” [10] The researcher must make specific assumptions about which types of spillovers are operative.
One can relax the non-interference assumption in various ways depending on how spillovers are thought to occur in a given setting.
One way to model spillover effects is a binary indicator for whether an immediate neighbor was also treated, as in the example above.
[11] The next step after redefining the causal estimand of interest is to characterize the probability of spillover exposure for each subject in the analysis, given some vector of treatment assignment.
Aronow and Samii (2017)[12] present a method for obtaining a matrix of exposure probabilities for each unit in the analysis.
First, define a diagonal matrix with a vector of treatment assignment probabilities
One important caveat is that this procedure excludes all units whose probability of exposure is zero (ex.
Figure 1 displays an example where units have varying probabilities of being assigned to the spillover condition.
Subfigure A displays a network of 25 nodes where the units in green are eligible to receive treatment.
In this case, subfigure B displays each node's probability of being assigned to the spillover condition.
Node 3 is assigned to spillover in 95% of the randomizations because it shares edges with three units that are treated.
When analyzing experiments with varying probabilities of assignment, special precautions should be taken.
These differences in assignment probabilities may be neutralized by inverse-probability-weighted (IPW) regression, where each observation is weighted by the inverse of its likelihood of being assigned to the treatment condition observed using the Horvitz-Thompson estimator.
[13] This approach addresses the bias that might arise if potential outcomes were systematically related to assignment probabilities.
The downside of this estimator is that it may be fraught with sampling variability if some observations are accorded a high amount of weight (i.e. a unit with a low probability of being spillover is assigned to the spillover condition by chance).
In some settings, estimating the variability of a spillover effect creates additional difficulty.
In order to conduct hypothesis testing in these settings, the use of randomization inference is recommended.
[15] This technique allows one to generate p-values and confidence intervals even when spillovers do not adhere to a fixed unit of clustering but nearby units tend to be assigned to similar spillover conditions, as in the case of fuzzy clustering.
