cumulative

D.3 Threats to internal validity

What the task asks: Recognize the uncontrolled factors, such as outside events and changes within the participant, that can make an intervention look effective when it was not, and know how designs guard against them.

Experimental Design5 min read5 practice questionsOfficial outline (opens in a new tab)

The big idea

A threat to internal validity is any factor other than the independent variable that could explain a change in behavior. A threat left uncontrolled becomes a confound, and the study can no longer claim that the intervention caused the change.

The classic list comes from group research (Campbell & Stanley, 1963), and most of it applies to single-case work as well. Some threats matter more in single-case designs, because participants are measured many times, conditions follow one another, and one person experiences every condition.

The exam usually describes a situation and asks you to name the threat. Sort them by what changed: the world, the person, the measurement or the order of conditions.

Breaking it down

Something in the world changed: history

A history threat is an event outside the study that happens at the same time as a phase change: a new teacher, a family move, a schedule change, an illness going around the classroom. It is the most common threat in applied settings, because real life does not pause for an experiment.

Something in the person changed: maturation

Maturation is change within the participant that comes with the passage of time: growing older, developing new skills, tiring over a long session, recovering from an illness. It is a particular risk in long studies with young children.

0102030BaselineIntervention123456789101112WeeksDifferent words used per session
What maturation looks like. Words used climb at the same steady rate in baseline and intervention, so growing older could explain the whole change. A baseline already trending in the expected direction is the warning sign.

Measuring changed the behavior: testing

A testing effect occurs when being measured repeatedly changes performance by itself. A learner may improve on a probe through practice alone, or act differently because they know they are being watched (reactivity). Single-case designs measure repeatedly, so this threat is always present. A steady baseline shows whether repeated measurement alone is producing a trend.

The measurement changed: instrumentation

An instrumentation threat comes from a change in the measurement system rather than the behavior. Observers may drift from the definition over time (observer drift, see C.8), a new observer may score differently, or observers who know the hypothesis may score in the expected direction. Interobserver agreement checks, retraining and observers who do not know the phase help.

Extreme scores drift back: regression to the mean

Regression to the mean is the tendency of unusually high or low measures to be followed by measures closer to the person’s average. If an intervention starts right after a run of unusually bad days, behavior may improve with no help from the intervention. The single-case remedy is to wait for steady-state responding before changing conditions.

024681012BaselineIntervention12345678910111213DaysTantrums per day
What regression to the mean looks like. Treatment starts right after two unusually bad days, and tantrums fall, but only back to the usual four or five a day. Waiting for steady baseline data would have avoided this false effect.

Earlier conditions affect later ones: multiple-treatment interference

When one person experiences several conditions, behavior in one condition can be shaped by experience in another. This is multiple-treatment interference, often discussed as sequence or carryover effects. It is a special concern in reversal designs with more than one treatment and in alternating treatments designs (see D.7). Counterbalancing the order and making each condition clearly distinct help. Sources classify this threat differently: Campbell and Stanley (1963) listed it under external validity, because a result found after several treatments may not hold when one is given alone, while single-case texts also treat it as a confound within the study.

Threats that mostly concern group designs

  • Selection: the groups differed before treatment began. Because each person in a single-case design is their own control, this threat is largely avoided.
  • Attrition: participants drop out, and those who leave differ from those who stay. It still matters in single-case research if participants who did not respond are left out of the report.
  • Diffusion of treatment: the intervention leaks into a condition that is supposed to be untreated. In single-case work this appears in multiple-baseline designs, for example when a teacher starts praising behavior in a setting still in baseline.

Treatment integrity belongs on this list too: if the IV is delivered poorly, the study tests something other than what it describes (see C.12).

Seeing it in practice

SituationThreat
Aggression drops the week intervention begins, which is also the week a classroom aide is added.History
A 3-year-old’s spoken vocabulary grows steadily across a six-month study, baseline included.Maturation
A student’s scores on a weekly spelling probe improve across baseline because the same words are tested each time.Testing
On-task scores rise after a new observer takes over, though video shows little change.Instrumentation
Intervention starts the day after the worst tantrum week of the year, and tantrums decline at once.Regression to the mean
In an alternating treatments comparison, Treatment B looks better only when it follows Treatment A.Multiple-treatment interference

Common mix-ups

  • History vs. maturation. History is an event outside the person. Maturation is a change within the person over time.
  • Testing vs. instrumentation. Testing means the behavior changed because it was measured. Instrumentation means the measurement changed while the behavior may not have.
  • Regression is not a treatment effect. It is a statistical tendency, controlled by not starting treatment on an extreme data point.
  • Treating one AB comparison as proof. A single change from baseline to intervention leaves history, maturation and regression open. The fix is replication: an outside event is unlikely to line up with every phase change in a reversal design, or with every staggered start in a multiple baseline.

Exam tip: Ask what else changed. An outside event points to history. Growing up or tiring points to maturation. Practice or being watched points to testing. A different observer or definition points to instrumentation. Starting after an extreme point points to regression. Earlier conditions affecting later ones point to multiple-treatment interference.

Quick recap

  • Threats to internal validity are alternative explanations for a change in the DV.
  • History (outside events) and maturation (change within the person) are the most tested.
  • Testing, instrumentation, regression and multiple-treatment interference matter especially in single-case work.
  • Selection and attrition are mainly group-design concerns; diffusion of treatment matters in multiple baselines.
  • Steady baselines, repeated demonstrations of effect, interobserver agreement and integrity checks are the main controls.

Further reading

  • Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Rand McNally.
  • Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). Applied behavior analysis (3rd ed.), chapter 10.
  • Kazdin, A. E. (2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). Oxford University Press.

Key terms

Internal validity
The extent to which a study shows that the independent variable, and not some uncontrolled factor, produced the change in the dependent variable. Designs that demonstrate experimental control have high internal validity.
Confounding variable (Confound)
An uncontrolled variable that changes along with the independent variable and may affect the dependent variable, so that its effect cannot be separated from the effect of the independent variable.
History effect (History)
A threat to internal validity in which an event outside the study, occurring at the same time as a change in conditions, produces or contributes to the change in behavior.
Maturation
A threat to internal validity in which changes within the participant that come with the passage of time, such as growing older, developing new skills, tiring or recovering from illness, produce the change in behavior.
Testing effect (Testing)
A threat to internal validity in which repeated measurement itself changes the participant's performance, through practice or through reactivity to being observed.
Instrumentation
A threat to internal validity in which the measurement system changes during a study, for example through observer drift, a new observer or a revised definition, so the data change even if the behavior does not.
Regression to the mean (Statistical regression)
The tendency of unusually high or low measures to be followed by measures closer to the person's average. It threatens internal validity when an intervention begins right after extreme data, because improvement may follow anyway.
Multiple-treatment interference (Sequence effects, Carryover effects)
A confound in which a participant's behavior in one condition is influenced by experience with another condition in the same study, either through the order of conditions (sequence effects) or through the lingering effects of an earlier condition (carryover effects). It is a concern in multielement designs and in reversal designs with more than one treatment.

Check yourself

5 questions on D.3. Choose an answer to see whether you're right and why.

  1. Question 1A behavior analyst introduces a self-management plan for Mateo's off-task behavior. The same week, Mateo's family hires a tutor who works with him every evening. Off-task behavior drops sharply. Which threat to internal validity is most clearly present?
    Show the answer

    Answer: B. The tutor is an outside event that began at the same time as the plan, so it could account for the change. That is history. Maturation would be a change within Mateo that comes from the passage of time, such as growing older.

  2. Question 2In a study of a client's vocal stereotypy, a new observer takes over data collection when intervention begins. Counts drop, but when the original observer scores videos of the same intervention sessions, that observer's counts stay at baseline levels. Which threat is most likely?
    Show the answer

    Answer: C. The change appears in the measurement, not in the behavior: two observers score the same sessions differently. That is an instrumentation threat. A testing effect would be a real change in the client's behavior caused by repeated measurement.

  3. Question 3A teacher refers Brayden, 9, after his worst week of disruptive behavior all year. The behavior analyst starts an intervention the next day, and disruptions fall back toward their usual level within days. Before concluding that the intervention worked, the analyst should consider:
    Show the answer

    Answer: C. Unusually extreme measures tend to be followed by measures closer to the person's average, with or without treatment. Starting right after the worst week, with no steady baseline, leaves this explanation open. Selection concerns differences between groups, which do not apply here.

  4. Question 4In an alternating treatments design comparing two prompting procedures, Rosa responds much better under the second procedure on days when it follows the first procedure than on days when it is run first. This pattern suggests:
    Show the answer

    Answer: A. Rosa's performance under one condition depends on her experience with the other condition, which is multiple-treatment interference, also called a sequence or carryover effect. It is not regression to the mean, because the difference follows the order of conditions rather than a return from one extreme score. Counterbalancing the order of conditions helps detect and control it.

  5. Question 5Why is selection less of a threat in single-case designs than in group designs?
    Show the answer

    Answer: B. Selection threats arise when the people in different groups differed before treatment. In a single-case design, each person is compared with their own baseline, so there are no separate groups to differ. Repeated measurement, not single measurement, is a feature of these designs.

Adds a step to your study record. Saved in this browser only.

Type to search all 104 guides and the glossary.