cumulative

I.7 Data-based decisions about supervision

What the task asks: Use data on supervisee performance, client outcomes, the integrity of the supervision itself and supervisee acceptance to judge whether supervisory practices are working, and change them when they are not.

Personnel Supervision and Management5 min read5 practice questionsOfficial outline (opens in a new tab)

The big idea

A behavior analyst would never leave a client’s intervention in place for months without looking at the data. Supervision deserves the same scrutiny. A supervisor may believe their approach works because supervisees seem happy and meetings feel productive, but a belief is not evidence.

Evaluating supervision means collecting data on whether it is changing what it is supposed to change, then using those data to keep, adjust or replace the approach. The Ethics Code makes this an obligation: behavior analysts continually evaluate their own supervisory practices using feedback from others and the outcomes of clients and supervisees, document those self-evaluations and adjust promptly when the results call for it (Standard 4.10, Evaluating Effects of Supervision and Training).

Breaking it down

What to measure

Supervision can be evaluated at several levels, and a strong evaluation uses more than one.

  1. Supervisee performance. Does the supervisee perform the targeted skills to criterion? Measures include integrity checklists scored during observation, competency-based assessment results (see I.4) and the quality of work products.
  2. Client outcomes. Are the supervisee’s clients making progress? This is the ultimate test. A supervisee with excellent integrity scores whose clients are not improving may be applying the right procedures to the wrong goals, or performing well only while watched.
  3. Integrity of supervision. Did the supervisor deliver supervision as planned? Procedural integrity usually describes how accurately a client intervention is carried out, but the idea applies to supervision too. Were the scheduled observations held? Were all the BST steps used? Was feedback specific and timely? When a supervisee is not progressing, check this first.
  4. Social validity. Do supervisees find the goals, procedures and results acceptable and useful? Social validity data come from short surveys, interviews or anonymous ratings, and they also show how the relationship is going (see I.2).
  5. Efficiency and retention. How long do supervisees take to reach mastery, and do they stay with the organization?

Performance monitoring

Performance monitoring is the ongoing, documented collection of data on supervisee performance through observation and structured evaluation. Standard 4.08 (Performance Monitoring and Feedback) calls for exactly this. Graph the data as you would client data, so trends are visible. A graph of a supervisee’s weekly integrity scores with a goal line makes decisions easy to see and easy to justify.

Making the decision

Use the same logic you would apply to client data (see H.7):

  • Steady progress toward criterion: continue, then fade the intensity of supervision as the skills maintain.
  • No progress after several observations: first confirm that supervision was delivered as planned. Then assess why the skill is not improving (see I.6) and change the procedure: add modeling, more rehearsal, more frequent feedback or a job aid.
  • Skill met in role-play but not in sessions: program for generalization with training and feedback in the real setting.
  • Supervisee skills improved but client outcomes did not: reexamine whether the targeted skills are the ones that matter for this client, and check performance when the supervisor is absent.
  • Good results but poor social validity: adjust the delivery (timing, format, tone) so the approach is sustainable.
0255075100BaselineVisits missedVisits as planned12345678910111213WeeksIntegrity (% of steps)
A technician's weekly integrity with a 90% goal line. Weeks with no point are planned observations that did not happen, and integrity stays flat near 65%. Restoring the planned schedule comes first, and scores then climb to the goal.

Single-case logic makes the evaluation stronger. A supervisor who introduces a new feedback procedure with one supervisee at a time, in a multiple-baseline design across supervisees, can show that the procedure, and not something else, produced the change.

050100DanaBaselineNew feedback procedure050100Luis050100Mei123456789101112WeeksIntegrity (%)
A multiple baseline across three supervisees. Each one's integrity changes only when the new feedback procedure reaches them. That staggered pattern is what shows the procedure, and not something else, produced the change.

Seeing it in practice

Data patternInterpretationSupervisory decision
A technician’s integrity rose from 60% to 92% after BST, and the client’s requests increasedSupervision is workingMaintain, then thin the observation schedule
Integrity has been flat at 65% for four weeks, and two of four planned observations were missedSupervision was not delivered as plannedRestore the planned schedule before changing the procedure
A trainee rates supervision highly, but competency scores have not changedSatisfying but not yet effectiveAdd rehearsal and feedback, and set measurable goals
Integrity is high when the supervisor is present and low on unannounced checksPerformance depends on the observerAdd self-monitoring and less predictable observations

Common mix-ups

  • Supervisee satisfaction is not effectiveness. Social validity matters, but it cannot replace performance and client data.
  • Supervisee performance is not the final outcome. Supervision exists to benefit clients, so check that better staff performance shows up as client progress.
  • Check the supervision before blaming the supervisee. If planned observations, BST steps or feedback were skipped, the supervisee never received the intervention.
  • Evaluation is continual. A single review at the end of supervision comes too late to help the supervisee or their clients.

Exam tip: When supervision data show no improvement, the first answer to consider is checking whether supervision was carried out as planned. The next is assessing the cause of the problem. Ending supervision or blaming the supervisee is almost never the first step.

Quick recap

  • Evaluate supervision with data, as you would any intervention.
  • Measure supervisee performance, client outcomes, the integrity of supervision, social validity and efficiency.
  • Graph supervisee data and use decision rules to continue, modify or fade.
  • Client outcomes are the ultimate test of supervision.
  • The Ethics Code requires continual, documented self-evaluation of supervisory practice (Standard 4.10).

Further reading

  • Sellers, T. P., Valentino, A. L., & LeBlanc, L. A. (2016). Recommended practices for individual supervision of aspiring behavior analysts. Behavior Analysis in Practice, 9(4), 274-286.
  • Turner, L. B., Fischer, A. J., & Luiselli, J. K. (2016). Towards a competency-based, ethical, and socially valid approach to the supervision of applied behavior analytic trainees. Behavior Analysis in Practice, 9(4), 287-298.

Key terms

Procedural integrity (Treatment integrity, Treatment fidelity, Implementation fidelity)
The extent to which a procedure is carried out exactly as designed. It is usually reported as the percentage of steps implemented correctly. Low integrity makes it impossible to tell whether an intervention works, and the same idea applies to whether supervision is delivered as planned.
Social validity
The extent to which the goals, procedures and outcomes of an intervention are acceptable, important and useful to the people it affects, such as clients, caregivers and supervisees. It is usually measured with surveys, interviews or rating scales.
Performance monitoring
The ongoing, documented collection of data on a supervisee's or employee's performance through direct observation, structured evaluations and review of work products, used to guide feedback and decisions about supervision.
Competency-based assessment
Assessment that measures whether a person can actually perform a skill to a specified standard, using direct observation, role-play or work products, rather than relying on time spent, attendance or self-report.

Check yourself

5 questions on I.7. Choose an answer to see whether you're right and why.

  1. Question 1A supervision plan calls for weekly observations with BST for a technician whose integrity is stuck at 65%. A review shows the supervisor held only two of the eight planned observations. What should the supervisor do first?
    Show the answer

    Answer: D. The technician received only a quarter of the planned supervision, so the data say nothing yet about whether BST works. Restoring the integrity of supervision comes before any other change. Switching methods or blaming the technician would be decisions about an intervention that never happened.

  2. Question 2A trainee gives supervision the highest possible rating on an anonymous survey, but the trainee's competency scores have not changed in two months. What is the best interpretation?
    Show the answer

    Answer: C. High social validity shows the trainee finds supervision acceptable, but the performance data show it is not changing the trainee's skills. Both kinds of data matter, and neither replaces the other. The supervisor should keep the parts of supervision the trainee values and change the procedures so the trainee's skills improve.

  3. Question 3A technician's integrity on a client's program has risen to 95%, but the client's skill acquisition has stalled for six weeks. What should the supervisor examine next?
    Show the answer

    Answer: D. Accurate implementation is not producing client progress, so the next question is whether the procedures and goals are right for this client. Supervision is judged ultimately by client outcomes. Chasing a few more points of integrity will not fix a plan that does not fit.

  4. Question 4Which data would best show that a new feedback procedure, rather than something else, improved supervisee performance?
    Show the answer

    Answer: A. When improvement follows the start of the procedure for each supervisee in turn, as in a multiple-baseline design, other explanations become unlikely. A single before-and-after comparison cannot rule out events that happened at the same time. Self-reports speak to acceptability, not to what caused the change.

  5. Question 5A technician's integrity is above 90% whenever the supervisor is in the room but drops to about 60% on unannounced checks. Which adjustment to supervision best fits these data?
    Show the answer

    Answer: D. The technician can do the skill, but performance depends on the supervisor's presence. Self-monitoring and unpredictable observations help bring performance under the control of the work itself rather than the observer. Retraining would teach a skill already in the repertoire, and announcing observations would make the pattern worse.

Adds a step to your study record. Saved in this browser only.

Type to search all 104 guides and the glossary.