INTRODUCTION
How can we know that a psychotherapeutic intervention is efficacious? How can we ascertain that positive effects observed following the delivery of a psychotherapeutic intervention are specifically attributable to the intervention itself? And, most importantly, can psychotherapy research be improved and to what extent?
Such questions are just some of the more challenging and intriguing issues that researchers involved with the investigation of psychotherapeutic interventions have handled in the last decades and are still handling today. If one takes into account the large number of available psychotherapies as well as the difficulties inherent in any attempt to properly conduct a psychotherapy study, it becomes evident that consistent effort should be directed towards the improvement of the methodological quality of studies designed to investigate the efficacy of psychotherapeutic interventions. This, in turn, could provide clinicians with more rigorous information about psychotherapy outcomes and could properly address several shortcomings that are frequently observed in current psychotherapy studies.
Accordingly, in this editorial I will highlight some of the most important critical issues that a well designed psychotherapy study should take into account and will build on my expertise into the field of mindfulness based interventions (MBIs) to show how such issues have been and can be successfully implemented in the design of future psychotherapy studies.
PSYCHOTHERAPY RESEARCH: WHAT SHOULD WE TAKE INTO ACCOUNT?
A thorough review of the large variety of methodological issues that could affect the results of a psychotherapy study is a huge matter that falls out with the aim and scope of this editorial. Rather, this paper aims to address some of the key issues that a psychotherapy study should take into account and suggests that the improvement of psychotherapy research is not only something that is largely needed but, more importantly, something that is feasible and should therefore be strongly encouraged.
The first question that a given psychotherapy study should address could be described as follows: how do we know that a specific intervention is efficacious for a given condition? A simple answer could be to deliver the intervention under investigation to a target population of subjects and to see if, by the end of the treatment period, some improvement, as measured with objective or subjective measures, can be observed. Although such an answer is somewhat intuitive and studies using an uncontrolled design have frequently been employed in psychotherapy studies, such a design does not allow control for important phenomena that could occur regardless of the administration of treatment. As Price and colleagues outlined in their seminal paper[1], the most common of such phenomena is the natural history of illness. Indeed, several conditions show a spontaneous improvement over time that can be unrelated to treatment. Furthermore, a second phenomenon that should be taken into account is the regression to the mean, a statistical phenomenon that assumes that individuals with extreme scores on any measure at one point probably will have less extreme scores, for purely statistical reasons, the next time they are tested.
How to deal with such issues? Two main approaches have usually been employed. The first one involves the comparison of the results of one’s own study with those reported in scientific studies focusing on untreated samples of subjects prospectively followed for a given period of time. The second approach involves the inclusion of a waiting list control group that receives no treatment. Although an empirical investigation aimed at comparing these two approaches in the field of psychotherapy research is still lacking, it is reasonable to suggest that the second approach carries the advantage of reducing possible sources of variance that could derive from the qualitative comparison of different populations by randomizing individuals to the treatment under investigation or to the waiting list (see also below).
Even though we exclude that the benefits related to treatment are not simply due to the natural history of illness or to the regression to the mean, a more important effect remains to be considered: the placebo or the “non specific effect” of treatment[1]. Over the last decades, the conceptualization of the placebo effect has shifted from the impossibility of the inert content of a placebo agent to produce clinically significant benefits to the concept of a simulation of an active therapy within a psychosocial context that would empower the influence of placebo[2]. The nature and the accurate description of the non specific effects of a given intervention represent a significant challenge for researchers involved in psychotherapy studies. Indeed, as several authors have recently underscored[3-5], in psychotherapy studies, the “placebo” control condition should be ideally matched as closely as possible with the intervention under investigation with regard to such non specific factors as benefit expectations, therapist contact, therapist (and, in some cases, group) support and educational information while, at the same time, it should exclude the “active ingredient(s)” of the specific intervention under investigation. Accordingly, it appears evident that, because a waiting list does not elicit any benefit expectation nor involves any educational information and therapist or group support, trials comparing a psychotherapeutic intervention with a waiting list control group cannot distinguish between the specific and the non specific effects of treatment (e.g.[6]).
A third issue that should be carefully considered in psychotherapy studies regards the random assignment of subjects to the treatment under investigation or to the control condition. Empirical evidence consistently supports the role of randomization in bias reduction. It has been shown, for instance, that nonrandomized trials are more likely to show advantage of an innovation over a standard treatment[7]. Furthermore, randomization procedures should be appropriate. Indeed, as Schulz et al[8] have stressed, only a few randomization procedures can be considered as appropriate and it is not surprising that appropriate randomization is one of the five criteria outlined in the Jadad Scale, one of the most widely used scales used to assess the quality of controlled trials thus far, to decide whether the quality of a given study can be considered as high or low[9]. An example of an appropriate randomization procedure is simple randomization, which is analogous to repeated fair coin-tossing. Such a procedure, although it represents the most basic of sequence generation approaches, is considered as significantly more reliable than other approaches, irrespective of their complexity and sophistication. If such a procedure cannot be successfully implemented, a blocked randomization, a procedure that controls the probability of obtaining an allocation sequence with an undesirable sample size imbalance in the intervention, can likewise be employed[8]. On the other hand, other procedures such as alternated allocation of patients should be considered as inappropriate because they carry a high risk of allowing the investigator anticipate which is going to be the following assignment and therefore to introduce a methodological bias.
In line with this point, allocation concealment should also be considered to ascertain that the methodological rigor of the randomization procedure is appropriately applied to a given study[10]. Indeed, without adequate allocation concealment, even random, unpredictable assignment sequences can be undermined. As an example, an analysis of 250 trials from 33 meta-analyses showed that randomized controlled trials in which treatment allocation was inadequately concealed, or in which concealment of allocation was unclear, yielded significantly larger estimates of treatment effects than those trials in which concealment was adequate[11]. As Schulz et al[10] outlined, many investigators involved with clinical trials can be tempted to decipher assignments, which, in turn, can subvert randomization. For some investigators implementing a trial, deciphering the allocation scheme might frequently become too great an intellectual challenge to resist. Therefore, methods that ensure appropriate allocation concealment should be implemented in future psychotherapy studies. One such example is the use of sealed envelopes numbered in advance, opened sequentially only after the participant’s name and other details are written on the appropriate envelope[12] and possibly containing cardboard or aluminum foils placed inside the envelope aimed at inhibiting the detection of assignments via hot lights.
A fourth important issue that should be taken into account is blinding. The rich history of blinding in clinical trials spans a couple of centuries[13]. However, significant misunderstandings exist with regard to a correct definition of blinding and consistent effort has recently been given to more properly define different types of blinding[14]. In extreme sum, in a double-blind design, currently considered as the most appropriate blinding methodology, investigators and assessors (frequently the same persons) as well as participants all remain unaware of the intervention assignments throughout the trial. However, several types of studies such as surgical intervention studies and psychotherapy studies cannot be double-blinded because of the difficulty of keeping subjects unaware of the intervention they are assigned to. Nevertheless, even though double blinding can be difficult if not even impossible to use in psychotherapy research, a single blind design in which at least the investigator is blind as to whether a given subject is receiving the intervention under investigation or the control intervention can be employed to reduce the risk of an assessment bias. In line with this view, several reviews currently assign one point of the Jadad Scale[9] when single blinding is employed (e.g.[6,15]).
Even though an appropriate control group as well as appropriate randomization and blinding procedures are employed, a challenging issue for psychotherapy studies is to ascertain that the intervention is appropriately delivered. First of all, this implies that the intervention should be manualized. Otherwise there would be no comparison to which the delivered intervention can be contrasted. Furthermore, it is also important to be able to measure the degree to which the intervention, as described in its treatment manual, is actually being administered. In other words, it is important to rely on adherence measures that offer a way of quantifying how faithfully the intervention has been provided[16] and whether the treatment has been successfully manipulated. This is usually achieved by means of audiotape or videotape recordings of the sessions and the use of adequate adherence scales through which an external evaluator expert in the treatment under investigation evaluates the extent to which the delivered intervention differs from the intervention described in the manual[17]. Finally, therapist experience should be considered as well. Indeed, although such a variable could have only a small effect on psychotherapy outcomes (e.g.[18,19]), it could nonetheless provide important complementary information that parallel the more “technical” information of treatment adherence[5].
Even when the issues mentioned above are appropriately addressed, the results of a psychotherapy study may still have limited usefulness if the sample size is not sufficiently powered to detect differences between groups (in superiority studies) or to ascertain that the apparent lack of difference between the intervention under investigation and the established treatment used as a comparison is not simply due to the lack of statistical power (in non-inferiority studies)[20]. In both cases, the authors should rely, whenever it is possible, on an effect size estimate based on prior studies dealing with the same or similar interventions for the intended clinical condition. Furthermore, several issues including the notion that in the forthcoming study, effect sizes could tend to the lower extreme of improvement, that a certain proportion of patients is likely to drop out over the study period and that for still other patients some information may not be appropriate or available, should also be considered in the design of a methodologically sound psychotherapy study (e.g.[21]).
In addition to the points outlined above, several further methodological issues should be considered.
As an example, there is consensus that for superiority trials, the intent-to-treat population (ITT) should be considered as the primary analysis population because it tends to avoid the over-optimistic estimates of efficacy that results from a per-protocol (PP) population that excludes subjects that for various reasons have dropped out from the intervention[22]. However, the choice of the appropriate analysis population in non-inferiority studies is far less defined. Although relying on the ITT population could be considered as a conservative approach even in this case, a simple simulation study aimed at investigating the degree of anticonservatism of the ITT population and to quantify the influence of non-compliers on the conclusion of a non-inferiority study found that, in the presence of non-compliers, the test for non-inferiority gives higher type I error rates (false positive findings) that increase with the proportion of non-compliers, and the degree of anticonservatism of ITT is inversely related to the size of the treatment effect in the non-complier group[23]. Therefore some authors have put forward that an hybrid ITT/PP analysis, which excludes non-compliant patients as in the PP analysis and properly addresses the impact of non-trivial missing data as in the maximum likelihood estimation-based ITT analysis, is a promising way of providing reliable non-inferiority tests (for a detailed description see[24]). Furthermore, the follow-up period should be consistent with that usually required to detect a significant effect of treatment on the target condition. In particular, the overall follow-up period should be based on existing literature focusing on a given psychotherapeutic intervention for a well specified clinical population and on the specific outcome under investigation (e.g. the reduction of acute depressive symptoms is supposed to require a shorter follow-up period in comparison with the prevention of future depression relapses). Finally, it is worth mentioning that authors other than the developers of the original psychotherapy program perform independent trials focusing on the efficacy of such interventions so as to provide evidence for treatment transportability and generalizability[25] and that large observational studies are performed in the community to ascertain intervention effectiveness. The distinction between efficacy and effectiveness is particularly important because, while efficacy measures how well a given intervention works in clinical trials, effectiveness relates to how well a treatment works in practice.
As we can see from this brief description, several issues should be considered in the design of a high quality psychotherapy study. In the next two sections I will briefly explore the concept of mindfulness and some of the main MBIs and will show how the methodological issues mentioned above have been successfully employed to improve current knowledge about such interventions.
MINDFULNESS BASED INTERVENTIONS
The word mindfulness derives from the Pali word sati, which can be found in early Buddhist scriptures such as the Abhidhamma[26], a classic scholastic compilation of Buddhist psychology and philosophy and, later, in the Vishuddimagga[27], a summary of the part of the Abhidhamma that deals with meditation. Because mindfulness concerns a clear awareness of one’s inner and outer experience, including thoughts, sensations, emotions, actions or surroundings as they exist at any given moment, in the Buddhist classical literature it has often been termed as “bare” attention[28-30] or alternatively as “pure” or “lucid” awareness[28,31,32], emphasizing that mindfulness is supposed to reveal what is occurring, before or beyond conceptual and emotional classifications about what is or has taken place. This, in turn, is supposed to reduce suffering related to the concept of an individual ego and ultimately lead to psychological well-being and happiness[33].
The cultivation of mindfulness has been a key element of several Buddhist meditations including Vipassana meditation[34] and Zen meditation[35] for centuries. More recently, the development of mindfulness has also proven to be a fruitful topic within clinical psychology[4]. Although there is not complete consensus as to how the concept of mindfulness should be properly defined and classified so far[36-39], mindfulness is currently conceptualized in psychological terms as a systematic development of attention to the present moment with a non-judgmental awareness of the inner and/or outer experiences. Kabat-Zinn[40], the founder of one of the most popular MBIs, as an example, describes mindfulness as the process of “paying attention in a particular way, on purpose, in the present moment and non-judgmentally” or, alternatively, as “the awareness that emerges through paying attention on purpose, in the present moment and non-judgmentally to the unfolding of experience moment by moment”[41].
MBIs, which include, among others, Mindfulness-Based Stress Reduction (MBSR)[42,43] and Mindfulness-Based Cognitive Therapy (MBCT)[44], have become a very popular form of treatment in contemporary psychotherapy as a means to deal with a large variety of physical, psychological and stress related problems[6,45-49]. Of note, it is worth mentioning that clinical findings are also increasingly supported by a large amount of objective neuropsychological and neurobiological findings[50,51].
In sum, MBSR is a standardized group-based meditation program conceived in the late ‘70s from the effort to integrate Buddhist mindfulness meditation with contemporary Western clinical and psychological practice[43,52]. MBSR is mainly based on three different techniques including (1) “body scan” which involves a gradual sweeping of attention through the entire body from feet to head, focusing non-critically on any sensation or feeling in body regions and using periodic suggestions of breath awareness and relaxation; (2) “sitting meditation” which involves both mindful attention on the breath or on the rising and falling abdomen, as well as on other perceptions, and a state of non judgemental awareness of cognitions and of the stream of thoughts and distractions that continuously flow through the mind; and (3) “Hatha yoga” practice which includes breathing exercises, simple stretches and posture designed to strengthen and relax the musculoskeletal system[43]. The standard program consists of 8 wk sessions with a duration of 2 and a half hours each and homework for 45 min a day, 6 d a week[43,52].
On the other hand, MBCT is a manualized 8 wk skills-training group program[44] based upon the theoretical framework of information processing theories[53] and integrating aspects of cognitive behavioral therapy for major depression (MD)[54] with components of the MBSR program developed by Kabat-Zinn[43]. MBCT was originally designed to teach patients in remission from recurrent MD to become more aware of, and to relate differently to, their thoughts, feelings and bodily sensations. An example includes recognizing thoughts and feelings as passing events in the mind rather than necessarily accurate readouts of reality. The original program teaches skills that allow individuals to disengage from habitual, automatic dysfunctional cognitive routines as a way to reduce future risk of relapses and recurrences of MD[44]. More recently, however, MBCT has also been successfully used for other clinical targets including, among others, the reduction of inter-episodic depression and anxiety levels in patients suffering from bipolar disorder[55,56] and the treatment of some anxiety disorders (e.g.[57-59]). In conclusion, MBIs can be described as psychological interventions whose purpose is to help patients achieve relief from such negative symptoms as chronic pain and depressive symptoms by targeting the extra baggage that is piled on to the symptoms in the form of, for example, negative thoughts and emotions by means of the development of an enhanced ability to cope with and/or relate differently to them.
MBIs AS AN EXAMPLE OF HOW PSYCHOTHERAPY RESEARCH MIGHT BE IMPROVED
As the field of mindfulness has grown exponentially in the last three decades in both quantity and complexity, it is well suited to show how the increasing sophistication of the methodological design can be successfully implemented in psychotherapy research and to highlight fruitful avenues for future research. Early studies focusing on the efficacy of MBSR for chronic pain patients mostly employed an uncontrolled design that did not distinguish between the specific effects of treatment, the non specific effects and the natural history of disease of such patients (e.g.[42,60]). Therefore, the only way observed findings could be critically evaluated was in a comparison between findings reported in the study and those usually observed in chronic pain patients under naturalist conditions. In the 1990s, the first studies appeared that compared MBSR with a waiting list control group to which subjects could be randomly (e.g.[61]) or non randomly assigned (e.g.[62]). Although the results were encouraging in that they suggested that subjects assigned to MBSR improved to a significantly higher extent than those assigned to the waiting list control group, such findings did not yet ascertain that benefits observed following MBSR could be specifically attributable to the interventions itself rather than to other non specific factors such as benefit expectations, group support, educational information and teacher’s care[47].
It is worth noting, however, that in more recent times several studies have been published that used appropriate comparison groups. One such example is the study published by Grossman and colleagues[63] comparing MBSR with a comparison group designed to match the non specific effects of MBSR while excluding the claimed “active ingredient”, i.e. mindfulness meditation practice. The control group employed by Grossman and colleagues included the presence of a trained, experienced group facilitator, participation in an 8 wk group setting of the same size and weekly format as the MBSR program, similar curriculum structure and equivalent amount of homework assignments, social support, relaxation training, gentle stretching exercises and weekly topical discussions. However, consistent emphasis was placed on not describing or training mindfulness skills to the control group. An even better design was subsequently employed by Zautra and colleagues[64]. The authors compared a MBI closely derived from MBSR with both an educational “non specific” control group and an active psychological control group (group cognitive behavioral therapy) in a sample of patients with rheumatoid arthritis. This design is particularly useful because, on the one hand it ascertains that both active treatments are significantly superior to the non specific comparison group and on the other hand, it investigates the existence of a possible specificity profile of active treatments that could be useful for future research. As an example, in the study by Zautra and colleagues[64], the authors found that mindfulness training was more efficacious for patients with rheumatoid arthritis and an history of MD while the cognitive behavioral intervention was more efficacious for patients with rheumatoid arthritis and without an history of MD.
With time, the improvement of randomization and blinding procedures has paralleled that of control groups employed in MBI research studies. Indeed, while the majority of early studies about MBIs employed an uncontrolled or a non randomized controlled design (e.g.[42,60,65], later studies have increasingly employed randomization, have properly described randomization procedures and have provided information about the appropriateness of allocation concealment[6]. A recent study investigating the efficacy of the adjunct of MBCT to treatment as usual (TAU) with TAU only for the prevention of MD relapses over a period of 1 year is a good example of the implementation of adequate randomization and blinding procedures to psychotherapy studies[66]. First of all, eligible subjects interested in MBCT were randomized to MBCT or to the waiting list control group using a stratified block randomization procedure. Stratification variables included site, number of previous depressive episodes and duration since remission from last episode. Secondly, they specified which strategy had been implemented to ensure adequate allocation concealment by stating that, after checking for inclusion and exclusion criteria and informed consent had been obtained, intervention was assigned to patients through sealed envelopes (Note, however, that information as to whether sealed envelopes contained cardboard or aluminum foils aimed at inhibiting detection of assignments was lacking).
Of note, the study by Bondolfi and colleagues[66], as well as many other ongoing (e.g.[21]) and recently published (e.g.[67]) studies, is also a good example of how sample size should be determined. Indeed, as the authors explained, sample size was estimated on the basis of previously reported differences of relapse rates between MBCT and waiting list control groups in MBCT studies. Additionally, an even better sample size estimate that has also taken into account the likelihood of drop outs has recently been described[68].
In the last decade, an increasing number of studies has also successfully controlled treatment adherence. In particular, several recent MBCT studies have reported that sessions were videotaped, that adherence to the MBCT protocol was assessed by experienced and independent MBCT therapists with a specific adherence scale (i.e. the Mindfulness Based Cognitive Therapy Adherence Scale[69]) and that treatment adherence could be considered at least as acceptable (e.g.[66,67]). Furthermore, the majority of recent MBCT studies consistently reported therapist experience and adherence to homework (for a review see[6]).
Notably, increasing attention has recently been given to the appropriateness of employed statistical analyses[21,68] and appropriate follow-up periods are increasingly being considered (e.g.[21]), even in short term studies (e.g.[70]). Finally, although large observational studies allowing for a proper evaluation of the effectiveness of MBSR and MBCT in the community are still lacking thus far, it is encouraging that an increasing number of studies performed by authors other than the developers of such interventions have recently been published that allow for an appropriate understanding of treatment transportability and generalizability (e.g.[66,71]).
CONCLUSION
Although the lack of a quantitative approach does not unequivocally evaluate whether and to what extent more recent studies exploring the usefulness of MBIs interventions for a large variety of clinical conditions have used a higher methodological quality as compared with older studies, a qualitative evaluation of the short review of studies mentioned above suggests that, with time, researchers concerned with MBIs are giving increasing attention to the methodological quality of their studies. Such observation is noteworthy because it suggests that improving psychotherapy research is feasible and should therefore be encouraged. Furthermore, with the increasing availability of psychotherapeutic approaches, increasing emphasis should be given to the methodological quality of future studies so as to provide clinicians with more rigorous information about psychotherapy outcomes and more reliable data that allows for a better understanding of which treatment could be best employed for a specific population of patients.
Of note, this does not criticize all studies that do not employ the methodological approaches mentioned above. As Orme-Johnson[5] has recently pointed out, whereas good randomized controlled trials may be the method of choice for demonstrating clinical efficacy, they may not be appropriate or may be too expensive to answer many other kinds of research questions. As an example, early pilot studies of a new psychotherapeutic approach could employ an uncontrolled design. If positive results are found, randomized controls should be performed to ascertain that positive effects observed in early studies are not only attributable to non specific factors of the intervention and to determine treatment transportability and generalizability. Such a claim is in line with the principles of Onken et al[25] who underscore that the development of new approaches should involve different progressive stages that guide the process of treatment development in a manner informed by ever more complex and rigorous tests of the novel protocol.
In conclusion, as the field of psychotherapy research moves forward, it will be increasingly important to use more rigorous methodological approaches. MBIs offer a good example of how psychotherapy research can be successfully improved. If any progress is to be achieved, the observations mentioned above could provide a precious source of information for the improvement of future psychotherapy studies.
Peer reviewer: Robin Alexander Emsley, Professor, Department of Psychiatry, Faculty of Health Sciences, Stellenbosch University, PO Box 19063, Tygerberg Campus 7505, Cape Town, South Africa
S- Editor Wang JL L- Editor Roemmele A E- Editor Zheng XM
