%0 Conference Proceedings
%A C Anna Spurlock
%A Peter Cappers
%A Ling Jin
%A Annika Todd-Blick
%A Patrick Baylis
%B 2016 ACEEE Summer Study on Energy Efficiency in Buildings
%D 2016
%G eng
%T Go for the Silver? Evidence from field studies quantifying the difference in evaluation results between “gold standard” randomized controlled trial methods versus quasi-experimental methods
%U http://aceee.org/files/proceedings/2016/data/papers/2_363.pdf
%X
Randomized controlled trials (RCTs) are widely viewed as the “gold standard” of evaluation. However, analysis of the effect of energy pricing has largely been conducted through quasi-experimental methods. Using a rare set of large-scale randomized field experiments of time-based electricity pricing, we compare the estimates obtained from commonly used non-experimental methods against RCT estimates. We demonstrate empirical evidence in favor of four stylized facts that highlight the importance of understanding two important sources of bias in this context: selection bias and spillover effects. First, difference-in-difference and propensity score methods tend to underestimate the true average treatment effect. Second, regression discontinuity methods tend to overestimate the effect. Third, selection biases in quasi-experimental methods tend to be more pronounced in opt-in treatments relative to opt-out treatments. Fourth, the three-in-five day baseline with an additive adjustment recomm ended by KEMA (2011) tends to underestimate the impact of the intervention, a pattern we attribute to intertemporal spillover effects.