cumulative

C.8 Validity and reliability of measurement

What the task asks: Judge whether a measurement system produces valid, accurate and reliable data, recognize what threatens each quality, and calculate and interpret interobserver agreement.

Measurement, Data Display, and Interpretation5 min read4 practice questionsOfficial outline (opens in a new tab)

The big idea

Decisions are only as good as the data behind them. Three qualities decide whether data can be trusted:

  • Validity: the data measure the behavior that matters, along the dimension that matters, at the times and places that matter.
  • Accuracy: the recorded values match what actually happened, the true values.
  • Reliability: measuring the same behavior again gives the same values.

Interobserver agreement (IOA), the match between two observers who record the same session independently, is the most common quality check in behavior analysis. It shows the data are believable, but it measures agreement, not truth. Two observers can agree perfectly and both be wrong.

Accurate and reliable True values that repeat Reliable, not accurate Repeat, but all miss Neither Miss and vary
Each dot is one measurement of the same session, and the bullseye is the true value. Reliable measures land close together; accurate ones land on the true value, so a tight cluster off center is reliable but not accurate.

Breaking it down

Threats to validity

  • Indirect measurement: recording a report or stand-in instead of the behavior (C.2).
  • Measuring the wrong dimension: counting tantrums when their length is the problem, or recording percentage correct when speed is the concern.
  • Measurement artifacts: results created by the measurement itself. Examples are discontinuous methods that distort the behavior (C.6), poorly scheduled observation (watching mornings when the behavior happens after lunch, or observing too rarely), and scales too limited to show change, such as a 10-item probe that cannot show growth past 10.

Threats to accuracy and reliability

Most are forms of human error:

  • Complex measurement systems: too many codes, behaviors or people for one observer.
  • Inadequate training: observers should practice on known samples until they meet a criterion before collecting real data.
  • Observer drift: a gradual, unintended change in how an observer applies the definition. Retrain and recalibrate periodically.
  • Observer expectations: knowing the hypothesis or the current condition can bias scoring. Use observers who do not know the condition where possible.
  • Observer reactivity: observers may record differently when they know their data are being checked. Unannounced checks help.

Reactivity also affects clients, who may behave differently while watched (C.2).

Checking the data

  • Accuracy: compare the observer’s data with true values, such as a video scored by an expert through careful, repeated review.
  • Reliability: have the observer rescore the same product or video and compare the two records.
  • IOA: two trained observers record the same session at the same time without seeing each other’s data.

Calculating IOA

DataMethodCalculation
CountsTotal countSmaller count ÷ larger count × 100
CountsMean count-per-intervalSmaller ÷ larger in each short interval, then average
CountsExact count-per-intervalIntervals with identical counts ÷ total intervals × 100
Discrete trialsTrial-by-trialTrials with agreement ÷ total trials × 100
TimingTotal durationShorter duration ÷ longer duration × 100
TimingMean duration-per-occurrenceShorter ÷ longer for each occurrence, then average
IntervalsInterval-by-intervalAgreements ÷ (agreements + disagreements) × 100
IntervalsScored-intervalAgreements on occurrence ÷ intervals either observer scored × 100
IntervalsUnscored-intervalAgreements on nonoccurrence ÷ intervals either observer left unscored × 100

Total count IOA is the most lenient: two observers who each count 20 hits get 100% even if they counted different hits. The per-interval methods are stricter, and exact count-per-interval is the strictest.

For interval data, interval-by-interval agreement is inflated when behavior is very rare or very frequent, because observers agree easily on the many empty (or full) intervals. Scored-interval IOA is the better check for low-rate behavior, and unscored-interval IOA for high-rate behavior. Some researchers also report Cohen’s kappa, which corrects for chance agreement.

Benchmarks

  • Agreement of at least 80% is the commonly cited minimum; 90% or higher is preferred.
  • Collect IOA in at least 20% of sessions, preferably 25 to 33%, spread across every condition and observer.
  • When IOA is low, clarify the definition and retrain. Do not simply average the two records.

Seeing it in practice

Two observers score Mei’s self-injury with 10-second partial-interval recording across 20 intervals. Both score occurrence in 3 intervals, they disagree on 2, and both leave the other 15 unscored.

  • Interval-by-interval IOA: (3 + 15) ÷ 20 × 100 = 90%.
  • Scored-interval IOA: 3 ÷ (3 + 2) × 100 = 60%.
Observer 1 Observer 2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Interval-by-interval: (3 + 15) ÷ 20 × 100 = 90% Scored-interval: 3 ÷ (3 + 2) × 100 = 60%, using only the outlined intervals
Mei's 20 intervals as each observer scored them (dark = scored). Interval-by-interval IOA counts the 15 empty intervals both observers agreed on. Scored-interval IOA keeps only the 5 outlined intervals, and the two disagreements (marked x) pull it down to 60%.

The 90% looks excellent, but it comes mostly from agreeing on empty intervals. On the intervals that matter, the observers agree only 60% of the time, so the definition needs work.

In supervision, a BCBA notices that two technicians who trained together have had 95% IOA on a learner’s aggression for months. Scoring a video carefully, the BCBA finds that both technicians recorded about 30% fewer hits than occurred: they had drifted together toward counting only hard hits. Their data agreed but were not accurate.

Common mix-ups

  • High IOA is not accuracy. Agreement can come from shared errors.
  • Accuracy and reliability differ. Reliable data repeat the same values; accurate data match the true values.
  • Interval-by-interval IOA can flatter rare behavior. Check scored-interval agreement too.
  • IOA from one condition only. It must be collected across every phase.
  • Observer drift is gradual. Initial training does not prevent it; ongoing checks catch it.

Exam tip: Know the formulas and do the arithmetic. For counts and durations, divide the smaller value by the larger. For intervals, check which intervals belong in the denominator: all of them, only scored ones, or only unscored ones.

Quick recap

  • Validity asks whether you measured the right thing; accuracy asks whether the values are true; reliability asks whether they repeat.
  • Validity threats: indirect measurement, the wrong dimension, measurement artifacts.
  • Accuracy and reliability threats: complex systems, poor training, observer drift, expectations, reactivity.
  • IOA formulas depend on the data type; total count is lenient, exact count-per-interval is strict.
  • Benchmarks: at least 80% (90% preferred), in 20 to 33% of sessions across all conditions.

Further reading

  • Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). Applied behavior analysis (3rd ed.), chapter 5.
  • Vollmer, T. R., Sloman, K. N., & St. Peter Pipkin, C. (2008). Practical implications of data reliability and treatment integrity monitoring. Behavior Analysis in Practice, 1(2), 4-11.

Key terms

Measurement validity (Validity of measurement)
The extent to which data reflect the socially significant behavior of interest, along the dimension that matters, under the conditions relevant to the question being asked. Indirect measurement, measuring the wrong dimension and measurement artifacts all threaten it.
Measurement accuracy (Accuracy of measurement)
The extent to which recorded values match the true values of the behavior as it actually occurred. Accuracy is checked by comparing data with true values obtained through special, more careful procedures; high interobserver agreement alone does not show it.
Measurement reliability (Reliability of measurement)
The consistency of a measurement system: the extent to which measuring the same behavior again produces the same values. Reliable data can still be inaccurate if the same error repeats each time.
Interobserver agreement (IOA)
The degree to which two or more observers who independently measure the same behavior report the same values. It is calculated with a formula suited to the data type, such as total count, interval-by-interval or total duration. IOA shows that data are believable, but high agreement does not prove they are accurate.
Observer drift
A gradual, unintended change in how an observer applies a measurement definition over time, so that the same behavior is scored differently later than it was at first. Periodic retraining and recalibration against scored samples catch it.
Reactivity
A change in behavior caused by the act of observing or measuring it. It applies to clients who behave differently while watched, and to observers whose recording changes when they know their data will be checked (observer reactivity).
Measurement artifact
A result that appears in the data only because of how the behavior was measured, not because of the behavior itself. Common sources are discontinuous methods that distort the behavior, poorly scheduled observation and measurement scales too limited to show change.

Check yourself

4 questions on C.8. Choose an answer to see whether you're right and why.

  1. Question 1Two observers count Rohan's out-of-seat behavior in the same 30-minute session. One records 16 instances and the other records 20. What is the total count IOA?
    Show the answer

    Answer: C. Total count IOA divides the smaller count by the larger: 16 ÷ 20 × 100 = 80%. Dividing the larger by the smaller (125%) is a common slip, and 20% is the difference between the counts as a share of the larger one. Total count IOA cannot show whether the observers counted the same instances.

  2. Question 2Two observers use partial-interval recording across 20 intervals to record a client's rare self-injury. Both score occurrence in 2 intervals, they disagree on 3, and both leave the other 15 unscored. What are the interval-by-interval and scored-interval IOA?
    Show the answer

    Answer: D. Interval-by-interval IOA uses every interval: the 2 shared occurrences plus the 15 shared nonoccurrences give (2 + 15) ÷ 20 = 85%. Scored-interval IOA uses only the 5 intervals that either observer scored: 2 ÷ 5 = 40%. The gap shows why scored-interval IOA is the better check for low-rate behavior.

  3. Question 3Two teachers who trained together have reached 96% IOA on a student's call-outs for two months. A researcher scoring video of the same sessions finds that both teachers recorded about a third fewer call-outs than occurred. What does this show?
    Show the answer

    Answer: C. The teachers agree closely, but both differ from the true values, so the data are not accurate. Observers who train together can drift in the same direction, which is why high IOA never proves accuracy. Reactivity would change the student's behavior, not create a gap between the teachers' records and the video of the same sessions.

  4. Question 4A team wants to know whether a social skills group helped Ethan make friends. They track his percentage correct on a worksheet about friendship rules. What is the main threat to the validity of these data?
    Show the answer

    Answer: A. Valid data measure the behavior the question is about. Knowing friendship rules on paper is a different behavior from talking and playing with peers, so this is indirect measurement. Nothing in the scenario points to agreement problems or drift, and the problem would remain even with perfect scoring.

Adds a step to your study record. Saved in this browser only.

Type to search all 104 guides and the glossary.