When most people think of “experiments,” they think of test tubes and telescopes, of Petri dishes and Bunsen burners. But the physical apparatus is not central to what an “experiment” means. Instead, what matters is the ability to specify different conditions–and then to observe how the differences in the underlying conditions alter the outcomes. When “experiments” are understood in this broader way, the application of “experiments” is expanded.
For example, back in 1881 when Louis Pasteur tested his vaccine for sheep anthrax, he gave the vaccine to half of a flock of sheep, expose the entire group to anthrax, and showed that those with the vaccine survived. More recently, the “Green Revolution” in agricultural technology was essentially a set of experiments, by systematically breeding plant varieties and then looking at the outcomes in terms of yield, water use, pest resistance, and the like.
This understanding of “experiment” can be applied in economics, as well. John A. List explains in “Field Experiments: Here Today Gone Tomorrow?” (American Economist, published online August 6, 2024). By “field experiments,” List is seeking to differentiate his topic from “lab experiments,” which for economists refers to experiments carried out in a classroom context, often with students as the subjects, and to focus instead on experiments that involve people in the “field”–that is, in the context of their actual economic activities, including work, selling and buying, charitable giving, and the like. As List points out, these kinds of economic experiments have been going on for decades. He points out that government agencies have been conducting field experiments for decades.
In Europe, the early social experiments in the late 1960s included electricity pricing schemes in Great Britain. In the US, social experiments can be traced to Heather Ross, an MIT economics doctoral candidate working at the Brookings Institution. The first wave of such experiments in the United States began in earnest in the late 1960s and included government agencies’ attempts to evaluate programs by deliberate variations in agency policies. Such large-scale social experiments included employment programs, electricity pricing, job training programs, and housing allowances. While this early wave of social experiments tended to focus on testing new programs, since the early 1980s major social experiments have examined various reforms that test incremental changes to existing programs. These experiments have had an important influence on policy, as they were recognized as contributing to the Family Support Act of 1988, which overhauled the AFDC program.
Again, the key to an “experimental” approach is to have control over the different conditions, and there’s control is more clear-cut in a test tube than in, say, people being charged for electricity in different ways. But as the advocates of field experiments point out, the approach doesn’t require that individual people be identical, nor that social interactions be like chemical reactions. If people are randomly divided into groups of sufficient size, then the groups will be broadly quite similar in underlying characteristics. This assumption of randomness needs to be questioned and checked, of course, and it turns out that some of the early experiments were not always truly random.
In addition, there is a question of “scalability,” and whether findings from a field experiment can be scaled up to a real-world program. As List has pointed out in earlier work, there is often a “voltage effect,” where a study finds an effect that doesn’t generalize to a broader population. A common issue here is that the details of the experiment may be hard to replicate at scale. As an example, List discusses what is called an “A/B experiment,” in this case an example where certain children got an intervention to prepare them for kindergarten, and others did not. He writes:
In the A/B experimental test of an early childhood program summarized … the program is found to triple Kindergarten Readiness: from 17% to 51%! One might view this result as extraordinary, and immediately want to scale the program. To understand why that choice is not prudent, consider what exactly we have learned from this research. If it is a typical social science experiment, then it has likely been conducted as an efficacy test: the “best-case” test of the program is arm B versus the control, arm A. To understand why more information is necessary, we must consider the incentives that the researchers faced. Those incentives are set up to create a petri dish that provides results that gives the intervention its “best shot,” or likewise the largest treatment effects. In this manner, we are answering the wrong question if we are attempting to provide policy advice. We are asking: can this idea work in the petri dish under the best-case situation rather than will this idea work at scale? This is the wrong question. We must not only do the efficacy test but also relevant tests of scale within the original discovery process. The economics of many situations demand such an approach.
List suggests that experiments shouldn’t be designed with the “petri dish” mindset, but instead can be designed to think in advance about “what constraints will the idea face at scale, what key factors can impact scaling.” No one said that experimental design was easy. List suggests a set of criteria for better and worse design–with the final column referring to studies that will inevitably bomb.
List suggest that when economists of the future approach a question like the productivity gains of a pin factory (Adam Smith’s famous example), they will do so with an experimental mindset, systematically varying the conditions to understand the outcomes. He writes:
In the past few decades, there is perhaps no empirical innovation that has changed economics more than field experiments. Via controlling the assignment mechanism, the experimenter sheds light on both the “effects of causes” and the “causes of effects”. Yet, the scientific insights do not end there. With some imagination and theoretical guidance, the experimenter can generate data that permits an informed prediction of whether the causal impacts of treatments implemented in one environment transfer to other environments, be them spatially, temporally, or scale differentiated. When these dual goals are achieved, the power of the experimental approach is unleashed.
