Variance Estimation in Evaluations With No

In experimental evaluations of policy interventions, the so-called Bloom adjustment is commonly used to estimate the impact of the treatment on the treated. It does so by rescaling the estimated impact of the intention to treat—that is, the overall treatment-control group difference in outcomes for the entire experimental sample—by the percentage of cases that took up the treatment offer. The practice of also rescaling the variance, as is common in the literature, imposes simplifying assumptions that may lead to biased variance estimates. We compare variances using the Bloom adjustment to variances that capture all the estimation error. While the difference between these variances is negligible in three experimental evaluations, we highlight three conditions that could result in larger relative bias: large impacts, large variability in compliance, and/or substantial endogeneity bias. The presence of these conditions could potentially result in a different conclusion for policy or practice