%0 Conference Proceedings %A C Anna Spurlock %A Peter Cappers %A Ling Jin %A Annika Todd-Blick %A Patrick Baylis %B 2016 ACEEE Summer Study on Energy Efficiency in Buildings %D 2016 %G eng %T Go for the Silver? Evidence from field studies quantifying the difference in evaluation results between “gold standard” randomized controlled trial methods versus quasi-experimental methods %U http://aceee.org/files/proceedings/2016/data/papers/2_363.pdf %X
Randomized controlled trials (RCTs) are widely viewed as the “gold standard” of evaluation. However, analysis of the effect of energy pricing has largely been conducted through quasi-experimental methods. Using a rare set of large-scale randomized field experiments of time-based electricity pricing, we compare the estimates obtained from commonly used non-experimental methods against RCT estimates. We demonstrate empirical evidence in favor of four stylized facts that highlight the importance of understanding two important sources of bias in this context: selection bias and spillover effects. First, difference-in-difference and propensity score methods tend to underestimate the true average treatment effect. Second, regression discontinuity methods tend to overestimate the effect. Third, selection biases in quasi-experimental methods tend to be more pronounced in opt-in treatments relative to opt-out treatments. Fourth, the three-in-five day baseline with an additive adjustment recomm ended by KEMA (2011) tends to underestimate the impact of the intervention, a pattern we attribute to intertemporal spillover effects.