Measures of skill at implementing a treatment

The evaluation of patient outcome

At first sight this is a compelling index as the goal of treatment is to benefit patients. In practice, however, it is problematic. Its main shortcoming is that it is an indirect measure as patient outcome is affected by variables other than the quality of the treatment provided, a key one being the characteristics of the patients in question. Patients vary in their responsiveness to treatment and unless this is taken into account when evaluating outcome data a false impression may be obtained. This is exemplified by the research on the use of mortality data to assess the competence of cardiac surgeons where it has emerged that most of the variability between surgeons is attributable to differences in the proportion of high risk patients treated.Therefore crude outcome data can be misleading, and this is especially the case if data are only available on a handful of patients, as is often the situation with psychological treatments. A treatment service may be evaluated in terms of patient outcome, but not individual courses of treatment or, indeed, individual therapists except in those unusual instances where there is high patient throughput as in certain treatment trials and implementation initiatives.

The evaluation of treatment sessions

A more widely used method for assessing the skill of a therapist is the evaluation of the quality of their treatment sessions (i.e., therapy quality is being used as an index of therapist competence). This therapy quality method requires that treatment sessions be evaluated using a standard procedure. In the field of cognitive behaviour therapy (CBT), for example, common practice is for treatment sessions to be rated using the Cognitive Therapy Scale or its revised version. These measures require that treatment sessions (usually recordings of them) be evaluated by a rater with respect to the presence and quality of certain therapist-determined features (e.g., the eliciting of key cognitions, the use of guided discovery, the setting of homework). On this basis, a score is generated and if it is above a specified threshold the session is judged to have been delivered sufficiently well (i.e).

In principle, this is an attractive way of evaluating therapist competence as it directly assesses the therapist’s performance at implementing a treatment. In practice, however, it is problematic. For example, it has proved difficult to define, operationalize and demarcate the aspects of treatment of interest with the result that inter-rater reliability has been less than satisfactory (e). Even more challenging has been the matter of validity. It has not been established that the various therapy rating scales measure what they purport to measure, nor has it been demonstrated that their threshold scores are appropriate ones for viewing sessions as having been delivered well enough.

There is another shortcoming. This concerns the features assessed. For example, the CTS and CTS-R focus largely on aspects of treatment that are common to most forms of CBT and, of necessity, ones that are expected to be present in most treatment sessions. They do not assess the disorder-specific strategies and procedures that are viewed as being central to the mode of action of most evidence-based forms of CBT. Thus they are more measures of the CBT “style” of conducting treatment than measures of any specific form of CBT.

An additional matter of importance is that there is rarely a formal protocol for the use of these instruments. Three matters are of concern. First, few sessions tend to be rated as doing so is time-consuming. As a result, generalizations are made based on limited data. This problem is compounded by the issue of patient variability mentioned earlier as it is easy to adhere to the treatment protocol with some patients whereas with others it is much more difficult. The second concern is that it is often the therapist who selects the sessions to be rated, a form of sampling that is highly likely to be biased. The third problem concerns the rater. Not infrequently ratings are made by someone who knows the therapist (such as the person providing supervision) and who might therefore already have a view about his or her ability, one which might colour their ratings.

The evaluation of standardised role plays

A third way of assessing therapists’ ability to implement treatments is through their performance in standardised role plays. This method is well established in medical education (where it is referred to as an “objective structured clinical examination” or OSCE .but it has only been employed to a limited extent to evaluate psychotherapeutic skills. This approach, however, has certain advantages over the therapy quality method of assessment, particularly as a means of evaluating the outcome of training. It will be discussed later.