Beyond the Lab: Quasi-Experimental Designs and Applied Research

(Goodwin, Ch. 10)

It is possible to conduct ‘true experiments’ in a field setting. Such studies do have external validity (generalizability) because one is usually attempting to generalize to non-lab settings. Participants in a field experiment do not usually realize that they are in a study. This eliminates possible problems such as participant bias or reactivity. However, it also creates ethical problems. A field experiment may also have problems of internal validity because it is often impossible to control for potentially confounding variables, i.e., weather, adjacency of other people, etc. It is often difficult to achieve randomization of subjects to the conditions. It may be difficult to measure the dependent variable in a reliable and valid manner.

Most field studies are ‘quasi-experiments’ – ‘sort-of’ experiments. Many are also applied research as opposed to basic research in the sense that they are trying to solve some immediate real-life problem rather than to test a more general theory of human behavior.

Consider a one condition study consisting of a pre-measure, an intervention, and a post-measure. Assume that there is a statistically significant change in the dependent variable between the two measures. Can one attribute the change to the intervention? Possible alternative explanations (called threats to the internal validity of this one condition pre-test—post-test design) include history, maturation, regression, testing, and instrumentation (instrument decay). Random assignment can eliminate these effects. They can, to some extent, be controlled for by conducting a quasi-experimental design and using a nonequivalent control group. However, even if the non-equivalent control group does not show a change from the pre-test to the post-test, the results may be linked to a ceiling effect, to participant bias (Hawthorne effect) or to in interaction between subject selection and, e.g., history or maturation.

An alternative to the non-equivalent control group design is an interrupted time series design in which the dependent variable is measured several times before the introduction of the intervention and for several times after it. Note (Goodwin p. 332-333; Figures 10.6 and 10.7) how one pattern strongly suggests that the intervention was effective and the problems with the other pattern of events.

Additions to the basic interrupted time series design involve: (1) the measurement of non-equivalent control groups as in the Connecticut speeding study; (2) the measurement of variables that are not expected to be affected by the intervention but might be affected by the various threats to internal validity; and (3) switching replications in which the intervention is repeated at a different time in a different place.

Program evaluation involves a needs analysis (is there a need for some ‘program’); formative evaluation (is the program being implemented as planned and used as expected); and a summative evaluation (an assessment of the program’s effectiveness).
Even if the program is effective, one should also conduct a cost-effectiveness analysis.

Analyses of the Drug Abuse Resistance Education Program (DARE) have concluded that it is not an effective program for reducing drug use amongst adolescents.