The big idea
A behavior analyst would never leave a client’s intervention in place for months without looking at the data. Supervision deserves the same scrutiny. A supervisor may believe their approach works because supervisees seem happy and meetings feel productive, but a belief is not evidence.
Evaluating supervision means collecting data on whether it is changing what it is supposed to change, then using those data to keep, adjust or replace the approach. The Ethics Code makes this an obligation: behavior analysts continually evaluate their own supervisory practices using feedback from others and the outcomes of clients and supervisees, document those self-evaluations and adjust promptly when the results call for it (Standard 4.10, Evaluating Effects of Supervision and Training).
Breaking it down
What to measure
Supervision can be evaluated at several levels, and a strong evaluation uses more than one.
- Supervisee performance. Does the supervisee perform the targeted skills to criterion? Measures include integrity checklists scored during observation, competency-based assessment results (see I.4) and the quality of work products.
- Client outcomes. Are the supervisee’s clients making progress? This is the ultimate test. A supervisee with excellent integrity scores whose clients are not improving may be applying the right procedures to the wrong goals, or performing well only while watched.
- Integrity of supervision. Did the supervisor deliver supervision as planned? Procedural integrity usually describes how accurately a client intervention is carried out, but the idea applies to supervision too. Were the scheduled observations held? Were all the BST steps used? Was feedback specific and timely? When a supervisee is not progressing, check this first.
- Social validity. Do supervisees find the goals, procedures and results acceptable and useful? Social validity data come from short surveys, interviews or anonymous ratings, and they also show how the relationship is going (see I.2).
- Efficiency and retention. How long do supervisees take to reach mastery, and do they stay with the organization?
Performance monitoring
Performance monitoring is the ongoing, documented collection of data on supervisee performance through observation and structured evaluation. Standard 4.08 (Performance Monitoring and Feedback) calls for exactly this. Graph the data as you would client data, so trends are visible. A graph of a supervisee’s weekly integrity scores with a goal line makes decisions easy to see and easy to justify.
Making the decision
Use the same logic you would apply to client data (see H.7):
- Steady progress toward criterion: continue, then fade the intensity of supervision as the skills maintain.
- No progress after several observations: first confirm that supervision was delivered as planned. Then assess why the skill is not improving (see I.6) and change the procedure: add modeling, more rehearsal, more frequent feedback or a job aid.
- Skill met in role-play but not in sessions: program for generalization with training and feedback in the real setting.
- Supervisee skills improved but client outcomes did not: reexamine whether the targeted skills are the ones that matter for this client, and check performance when the supervisor is absent.
- Good results but poor social validity: adjust the delivery (timing, format, tone) so the approach is sustainable.
Single-case logic makes the evaluation stronger. A supervisor who introduces a new feedback procedure with one supervisee at a time, in a multiple-baseline design across supervisees, can show that the procedure, and not something else, produced the change.
Seeing it in practice
| Data pattern | Interpretation | Supervisory decision |
|---|---|---|
| A technician’s integrity rose from 60% to 92% after BST, and the client’s requests increased | Supervision is working | Maintain, then thin the observation schedule |
| Integrity has been flat at 65% for four weeks, and two of four planned observations were missed | Supervision was not delivered as planned | Restore the planned schedule before changing the procedure |
| A trainee rates supervision highly, but competency scores have not changed | Satisfying but not yet effective | Add rehearsal and feedback, and set measurable goals |
| Integrity is high when the supervisor is present and low on unannounced checks | Performance depends on the observer | Add self-monitoring and less predictable observations |
Common mix-ups
- Supervisee satisfaction is not effectiveness. Social validity matters, but it cannot replace performance and client data.
- Supervisee performance is not the final outcome. Supervision exists to benefit clients, so check that better staff performance shows up as client progress.
- Check the supervision before blaming the supervisee. If planned observations, BST steps or feedback were skipped, the supervisee never received the intervention.
- Evaluation is continual. A single review at the end of supervision comes too late to help the supervisee or their clients.
Exam tip: When supervision data show no improvement, the first answer to consider is checking whether supervision was carried out as planned. The next is assessing the cause of the problem. Ending supervision or blaming the supervisee is almost never the first step.
Quick recap
- Evaluate supervision with data, as you would any intervention.
- Measure supervisee performance, client outcomes, the integrity of supervision, social validity and efficiency.
- Graph supervisee data and use decision rules to continue, modify or fade.
- Client outcomes are the ultimate test of supervision.
- The Ethics Code requires continual, documented self-evaluation of supervisory practice (Standard 4.10).
Further reading
- Sellers, T. P., Valentino, A. L., & LeBlanc, L. A. (2016). Recommended practices for individual supervision of aspiring behavior analysts. Behavior Analysis in Practice, 9(4), 274-286.
- Turner, L. B., Fischer, A. J., & Luiselli, J. K. (2016). Towards a competency-based, ethical, and socially valid approach to the supervision of applied behavior analytic trainees. Behavior Analysis in Practice, 9(4), 287-298.
Key terms
- Procedural integrity (Treatment integrity, Treatment fidelity, Implementation fidelity)
- The extent to which a procedure is carried out exactly as designed. It is usually reported as the percentage of steps implemented correctly. Low integrity makes it impossible to tell whether an intervention works, and the same idea applies to whether supervision is delivered as planned.
- Social validity
- The extent to which the goals, procedures and outcomes of an intervention are acceptable, important and useful to the people it affects, such as clients, caregivers and supervisees. It is usually measured with surveys, interviews or rating scales.
- Performance monitoring
- The ongoing, documented collection of data on a supervisee's or employee's performance through direct observation, structured evaluations and review of work products, used to guide feedback and decisions about supervision.
- Competency-based assessment
- Assessment that measures whether a person can actually perform a skill to a specified standard, using direct observation, role-play or work products, rather than relying on time spent, attendance or self-report.
Check yourself
5 questions on I.7. Choose an answer to see whether you're right and why.
Show the answer
Answer: D. The technician received only a quarter of the planned supervision, so the data say nothing yet about whether BST works. Restoring the integrity of supervision comes before any other change. Switching methods or blaming the technician would be decisions about an intervention that never happened.
Show the answer
Answer: C. High social validity shows the trainee finds supervision acceptable, but the performance data show it is not changing the trainee's skills. Both kinds of data matter, and neither replaces the other. The supervisor should keep the parts of supervision the trainee values and change the procedures so the trainee's skills improve.
Show the answer
Answer: D. Accurate implementation is not producing client progress, so the next question is whether the procedures and goals are right for this client. Supervision is judged ultimately by client outcomes. Chasing a few more points of integrity will not fix a plan that does not fit.
Show the answer
Answer: A. When improvement follows the start of the procedure for each supervisee in turn, as in a multiple-baseline design, other explanations become unlikely. A single before-and-after comparison cannot rule out events that happened at the same time. Self-reports speak to acceptability, not to what caused the change.
Show the answer
Answer: D. The technician can do the skill, but performance depends on the supervisor's presence. Self-monitoring and unpredictable observations help bring performance under the control of the work itself rather than the observer. Retraining would teach a skill already in the repertoire, and announcing observations would make the pattern worse.
Adds a step to your study record. Saved in this browser only.