This article has Open Peer Review reports available.
Guided online treatment in routine mental health care: an observational study on uptake, drop-out and effects
© Kenter et al.; BioMed Central Ltd. 2013
Received: 2 February 2012
Accepted: 28 January 2013
Published: 31 January 2013
Due to limited resources patients in the Netherlands often have to wait for a minimum of six weeks after registration for mental health care to receive their first treatment session. Offering guided online treatment might be an effective solution to reduce waiting time and to increase patient outcomes at relatively low cost. In this study we report on uptake, drop-out and effects of online problem solving treatment that was implemented in a mental health center.
We studied all 104 consecutive patients aged 18–65 years with elevated symptoms of depression, anxiety and/or burnout who registered at the center during the first six months after implementation. They were offered a five week guided online treatment. At baseline, five weeks and twelve weeks we measured depressive (BDI-II), anxiety (HADS-A) and burnout symptoms (MBI).
A total of 55 patients (53%) agreed to start with the online treatment. Patients who accepted the online treatment were more often female, younger and lower educated than those who refused. There were no baseline differences in clinical symptoms between the groups. There were large between group effect sizes after five weeks for online treatment for depression (d = 0.94) and anxiety (d = 1.07), but not for burnout (d = −.07). At twelve weeks, when both groups had started regular face-to-face treatments, we no longer found significant differences between the groups, except for anxiety (d = 0.69).
The results of this study show that the majority of patients prefer online guided online treatment instead of waiting for face-to-face treatment. Furthermore, online PST increases speed of recovery and can therefore be offered as a first step of treatment in mental healthcare.
Depression and anxiety disorders are common mental disorders with a lifetime prevalence of 20% for depression, and 16% for anxiety disorders . This results in loss of quality of life and substantial costs for the individual as well as for the society [2, 3]. Various effective face-to-face treatments are available . However, due to economic reasons there is a growing demand for shorter and less expensive treatments. Use of guided self-help treatments have been frequently suggested as one of the solutions. Self-help is defined as “a psychological treatment in which the patient takes home a standardized psychological treatment protocol and works through it independently” . Guided self-help aims at helping patients to understand treatment techniques, to motivate patients and to responds to difficulties the patient may confront when working with self-help material. This might be provided by email, websites, or telephone . Randomized controlled trials (RTC’s) have shown that guided self-help treatments are effective in reducing symptoms of depression and anxiety, regardless of the format in which it is offered (book or Internet) [7–9].
The most well-known example of integrating self-help in everyday mental health care is the Improving Access to Psychological Therapies (IAPT) initiative in the UK, which offers low to high-intensity treatments according to a stepped care model. First patients with mild to moderate anxiety and depression receive guided self-help, delivered by psychological wellbeing practitioners. After that, patients in need of more treatment have access to more intense psychological interventions. Results show good outcomes . The two year data show large uncontrolled effect sizes for depression (1.07, 95% CI: 0.88-1.29) and for anxiety (1.04, 95% CI: 0.88-1.23) with reliable and clinically significant change in 55% of cases .
In sum, from RCT’s we know that Internet guided self-help is effective and IAPT shows that guided self-help can be implemented effectively in routine care. But we lack data on implementation of Internet based treatment in routine mental healthcare.
In this study we used data of a mental health institution that implemented Internet treatment in standard care and offered it during the waitlist period, which is the time between application and the first face-to-face session.
We report on (1) the uptake of the Internet treatment, which is defined as the percentage of patients who accept the offer of online treatment during waiting for face-to-face treatment and the percentage of drop-out, (2) the demographic and clinical profile of patients who prefer online treatment, and (3) the symptom reduction due to the online treatment compared to the symptom reduction in the group of patients who declined the offer and decided to wait for regular face-to-face sessions.
All eligible patients who registered for help at a mental health service were offered online treatment before the onset of face-to-face treatment. Participants who were willing to use the five week online treatment could start with the treatment the same day they signed up for the study. The other participants started with five weeks of waiting. They were asked by phone why they declined to offer of online treatment. After five weeks, both groups received the regular face-to-face treatment.
The internet course was not offered to every patient but only to those 18 years or older, with elevated depressive symptoms (scoring ≥14 on the BDI-II), anxiety symptoms (scoring > 8 on the HADS) and/or work-related stress (scoring ≥ 2.2 on high emotional exhaustion in combination with ≤ 2.2 depersonalization or ≤ 3.66 personal accomplishment on the MBI). It was not offered to patients with suicidal thoughts, without Internet access, or without sufficient understanding of the Dutch language. All new eligible patients of the mental health care center were informed about the Internet course and the study, by e-mail and/or by phone. All participating patients gave written informed consent to use the routine outcome monitoring data for this study. The enrollment took place between October 2010 and April 2011.
The online treatment program that was used in this study was a five week Problem Solving Treatment (PST) [8, 12]. It has proved to be effective in reducing symptoms of depression and anxiety [8, 9]. Also, the generic character of the online PST makes it suitable to address different kinds of symptoms. This is useful because of the high prevalence of comorbidity in psychiatric disorders [3, 13] and can therefore fit well into a stepped-care model in specialized or primary mental health care institutions .
The online treatment has three steps. First, participants have to write down things that really matter to them. Second, they have to describe their current problems and subdivide these into unimportant problems (i.e. unrelated to the things that matter to them), solvable problems, and unsolvable problems (e.g. the death of a loved one). For each category a different technique is suggested to solve or cope with the problems. The core of the program consists of solving the solvable problems. This is done in six steps: (1) writing a full description of the problem, (2) generating multiple solutions, (3) selecting the best solution, (4) making a plan for carrying out the solution, (5) actually carrying out the solution and (6) evaluating whether the solution has solved the problem. During the last step participants create a plan for the future in which they write down how they will try to achieve the things that really matter to them. The course takes five weeks and consists of one lesson a week. The website of the course contains instructions, exercises, examples of people applying the principles of PST. All participants received support in the form of feedback from a trained Master student in Clinical Psychology. The feedback was given within three working days by email and consisted of guiding the participants through the course and more personalized feedback (e.g. psycho education on how to cope with depression, anxiety and/or stress). The time spent on feedback was approximately 40 minutes per lesson.
All participants were asked to fill out the questionnaires at baseline, five weeks later (or directly after the online treatment) and twelve weeks after baseline. A week prior to an assessment point, participants received an email alert. The self report questionnaires were administered online.
Depressive symptoms were measured with the 21-item Beck Depression Inventory Second Edition (BDI-II) , a widely used multiple-choice self-report inventory which detects, assesses, and monitors changes in the severity of depressive symptoms as listed in the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders Fourth Edition (DSM-IV; 1994). It consists of questions with answers being scored on a 0–3 scale, about how the person felt the last week. The total score varies between 0 and 63 with higher scores indicating more depression. Scores below 14 indicate absence of depressive symptoms. We used this cut-off score as an indication for recovery from depressive symptoms. The validity of the BDI-II has been tested in different populations . In our study the Cronbach’s alpha was .86.
For measuring anxiety symptoms the 7-item anxiety subscale of the Dutch version of the Hospital Anxiety and Depression Scale (HADS) was used . Item-responses are on a 0 to 3 scale. The total score varies from 0 to 21, with higher scores indicating greater degrees of symptom severity. The cut-off score of 8 is an inclusion criterion for this study. The Dutch version of the HADS showed good homogeneity and reliability . In our study the Cronbach’s alpha was .55 for the anxiety subscale.
The Dutch version of the Maslach Burnout Inventory (MBI) was used to measure work related stress. It consists of 15 items and assesses three components: emotional exhaustion (EE), depersonalisation (D), and personal accomplishment (C). The items are written in the form of statements about personal feelings of attitudes and are answered on 7-point scale (ranging from 0, “never” to 6 “very often”). Higher scores on the two subscales emotional exhausting and depersonalisation indicate more experienced work-related stress, while lower scores on personal accomplishment corresponds to higher degrees of work-related stress. A score >2.2 on the subscale emotional exhaustion in combination with a score >2.2 on the subscale depersonalisation or a score <3.66 on personal accomplishment is an indication for a burnout syndrome. Its validity and reliability has been tested in different populations [17, 18]. In our study the Cronbach’s alphas for the subscales emotional exhausting, depersonalisation and personal accomplishment were .91, .85 and .87.
Quality of life
We used the sixth item of the EuroQol questionnaire (EQ-5D) to assess quality of life . Participants are asked to rate their current health from 0 (worst imaginable) to 100 (best imaginable).
Statistical analyses were conducted using SPSS 18 for Windows. Baseline differences in demographics and clinical characteristics between the two groups were investigated using Chi-square and independent samples t-tests. We examined the effects between the two groups on the three outcome measures (depression, anxiety and work related stress) separately. First, we used ANOVA and follow-up t-tests to evaluate differences between the two groups for each assessment point. Second, we used independent samples t-tests to investigate the difference in mean scores between the two groups. Third, we calculated between group effect sizes (Cohen’s d) by subtracting the post-test mean score of the WL group from the post-test mean score of the PST group, divided by the pooled standard deviation. Effect sizes larger than d = 0.8 were considered to be large, d = 0.5–0.8 moderate, and d = 0.2–0.5 small . Fourth, clinical significant change was determined with norms for the outcome measure. We calculated differences between the two groups in the percentage of patients who had recovered. We considered participants to have recovered when they scored <14 on the BDI-II, < 8 on the HADS-A and < 2.2 on emotional exhaustion in combination with a score < 2.2 on depersonalisation or > 3.66 on personal accomplishment on the MBI. Improvement was determined following the suggestions of Jacobson and Truax  calculating a reliable change index (RC).
All analyses were performed on the intention-to-treat sample. Pre-test data were available for all participants. Missing values of post-test non-responders were imputated by multiple regression analyses using available baseline data, demographics as well as data on baseline severity, from all participants.
Ethical approval and registration was not required for this study according to Dutch law.
Uptake of the course and drop-out
At five weeks 12.5% (14.5% PST and 10.2% WL; p = .48) of the sample failed to return the study questionnaire, and this was 29.8% at twelve weeks (29.1% PST and 30.6% WL, p = .63). Reasons for dropout were unknown. Analyses show no significant differences in clinical or demographical characteristics between responders and non-responders.
Description of the sample at baseline
Demographics at baseline
All (N = 104)
PST (n = 55)
WL (n = 49)
Age in years (SD)
t(102) = −4.81, p < .001
χ 2 (1) = 16.27, p < .001
χ 2(2) = 8,62, p = .01
Baseline differences and clinical outcomes after 5 weeks
The mean scores on depression, anxiety, burnout and quality of life were calculated for both groups at each assessment (see Table 2).
Clinical outcomes after 12 weeks
Clinically significant change
At baseline 17 participants in the PST group and 16 participants in the WL group fulfilled the criteria for burnout. Although there is a significant decrease in the number of people with burnout in the full sample (PST; n = 7, WL; n = 5), there are no significant differences between the two groups at five weeks (χ2(1) = .17, p = .68) and twelve weeks (χ2(1) = .39, p = .53).
At 5 weeks reliable improvement was 52.7% (n = 29) for depression, and 40% (n = 22) for anxiety for the PST group. For the waitlist group reliable improvement at 5 weeks was 20.4% (n = 10) for depression, and 14.3% (n = 7) for anxiety (depression: χ2 (1,104) = 11.55, p = .001; anxiety: χ2(1,104) = 8.52, p = .004). Reliable deterioration was 1% or less. Results show significant between-group differences in terms of clinically significant change on the BDI-II and HADS-A. Recovery (i.e. reliable improvement from a pretest score above cut-off to a post-test score below cut-off) occurred more often in the PST group (n = 28, 50.1%) after 5 weeks than in WL (n = 10, 20.4%) for depression (χ2(1,104) = 10.40, p = .001). This was also true for anxiety (PST; n = 10, 18.2%, WL; n = 3, 6.1%, χ2(1,104) = 3.45, p = .05).
The number of participants showing clinical significant change at 12 weeks for depression were n = 21 for PST and n = 17 for WL (χ2 (1,104) = .225, p = .635), and for anxiety n = 46 for PST and WL n = 29 (χ2 (1,104) = 7.705, p = .006).
Results are from analyses performed on intention to treat data. We also conducted the same analyses on the observed data from completers only. Results show no significant differences in comparison with intention to treat analyses.
This study shows that Internet treatment in routine mental health care is feasible. About 50% of all patients preferred to carry out Internet treatment instead of waiting for face-to-face treatment. They were younger, more often female and more often lower educated than the patients who did not want to do this. Reasons patient chose to wait instead of carrying out the online treatment were the fear that personal information was not save on the Internet, the wish of talking to a person instead of a computer, and the believe of not having sufficient computer skills. The adherence to treatment was low, about two-thirds completed a substantial part of the online treatment (3 lessons or more), but only 18% carried it out completely. Several explanations for these outcomes can be suggested: it can be either due to characteristics of the patient sample, the way of treatment delivery, or the content of the online course itself which after three lessons doesn’t significantly differ from the previous lessons.
There were no significant differences between the two groups in clinical characteristics at baseline. Results show that patients who chose to carry out online PST improved more than patients who chose to stay on the waiting list. The online treatment had large between-group effect sizes after five weeks for depression, anxiety and quality of life. After five weeks both groups started with face-to-face treatment. The differences between the two groups then disappeared, although after twelve weeks the online PST group still outperformed the WL group for anxiety symptoms. Furthermore, 34% of the participants in the online PST group recovered to a clinically significant degree at five weeks while in the WL group only 9.1% had recovered after five weeks. Regarding burnout symptoms, online PST lead to a decrease in symptoms, but so did the control group. There were no significant differences between the two groups regarding burnout scores.
The online PST group showed rapid improvements after five weeks. Improvement directly after the start of treatment is not an unfamiliar finding . Explanations for this effect could be due to non-specific factors like treatment expectations or perhaps the focus at the beginning of the online PST which consists of problem-solving . In this study we found reductions of symptoms of depression, anxiety and burnout for the online PST group but also for the WL group. Perhaps the reduction of symptoms in the WL group was caused by test procedures (e.g. questionnaires) or spontaneous improvement. Improvement of the control group is common and also seen in other studies [24, 25]. However, the outcomes of this study clearly show that participants who received online PST improved more.
Notable is the fact that despite the high recovery rates, all participants started face-to-face treatment after five weeks. Internet interventions are often considered as a first step in a stepped care model assuming that after this first step, not everybody needs a more intensive step, like face to face treatment. However, in this study the first face-to-face treatment session was already scheduled before start of the online treatment and the necessity of this appointment was not checked. We did not register the further uptake of treatment sessions.
Limitations and future research
There are limitations of this study that should be noted. First, we only obtained data from one mental health care service. Second, we have no data available on why patients chose the online treatment over waiting. Perhaps these patients were more motivated and more ready to change. These factors, and also expectations of the PST intervention may be associated with the uptake and effects, instead of the intervention itself. Additional studies are required to examine patients motivation, and if the results hold for other mental health care institutions with patients from different ethnic backgrounds and socioeconomic groups.
Another limitation concerns the lack of diagnosis according to DSM-IV criteria. Online self-report was used to include participants. Even though we can assume that only patients with a DSM diagnoses were referred to the service by their GP, this is not certain. With regard to psychometric properties, the questionnaires used in this study were not tested for online use. Also, the HADS-A had a low alpha in our sample. The psychometric properties can be different from the paper versions but can also lead to more discloser of sensitive information in comparison with face-to-face interviews .
In this study, we were challenged with attrition and low adherence. Only 18% of the participants completed the whole online PST and 65% completed at least three sessions. Future research could focus on ways to minimize attrition and increase adherence, because adherence is correlated with outcome in cognitive therapy . Also, we lack data on how many therapy sessions patients received after the five weeks of PST or waiting. Future research could evaluate the of cost effectiveness for the intervention and examine if patients require less face to face treatment after completing an Internet intervention.
In sum, the results of this study show that the majority of patients prefer online guided treatment instead of waiting for face to face treatment. For symptoms of depression and anxiety online PST increases speed of recovery. Therefore, we recommend it to be used more broadly as a first step of treatment in mental health care.
This study was performed at the BOSgroep, a mental health care institution for psychotherapy, research and coaching, located in Badhoevedorp and Heemstede, in the Netherlands.
- de Graaf R, de Graaf R, ten Have Mt, van Dorsselaer S: De psychische gezondheid van de Nederlandse bevolking. NEMESIS-2: Opzet en eerste resultaten. 2010, Utrecht: Trimbos-instituut UtrechtGoogle Scholar
- Smit F, Cuijpers P, Oostenbrink J, Batelaan N, de Graaf R, Beekman A: Costs of nine common mental disorders: implications for curative and preventive psychiatry. J Ment Health Policy Econ. 2006, 9: 193-200.PubMedGoogle Scholar
- Kessler RC, Chiu WT, Demler O, Merikangas KR, Walters EE: Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005, 62: 617-627. 10.1001/archpsyc.62.6.617.View ArticlePubMedPubMed CentralGoogle Scholar
- Cuijpers P, van Straten A, Andersson G, van Oppen P: Psychotherapy for depression in adults: a meta-analysis of comparative outcome studies. J Consult Clin Psychol. 2008, 76: 909-922.View ArticlePubMedGoogle Scholar
- Cuijpers P, Schuurmans J: Self-help interventions for anxiety disorders: An overview. Current Psychiatry Reports. 2007, 9: 284-290. 10.1007/s11920-007-0034-6.View ArticlePubMedGoogle Scholar
- van ’t Hof E, Cuijpers P, Stein DJ: Self-help and internet-guided interventions in depression and anxiety disorders: a systematic review of meta-analyses. CNS Spectrums. 2009, 14: 34-40.Google Scholar
- Andersson G, Carlbring P, Berger T, Almlov J, Cuijpers P: What Makes Internet Therapy Work?. Cogn Behav Ther. 2009, 38: 55-60. 10.1080/16506070902916400.View ArticlePubMedGoogle Scholar
- van Straten A, Cuijpers P, Smits N: Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res. 2008, 10: e7-10.2196/jmir.954.View ArticlePubMedPubMed CentralGoogle Scholar
- Warmerdam L, van Straten A, Twisk J, Riper H, Cuijpers P: Internet-based treatment for adults with depressive symptoms: randomized controlled trail. J Med Internet Res. 2008, 10: 44-10.2196/jmir.1094.View ArticleGoogle Scholar
- Clark DM, Layard R, Smithies R, Richards DA, Suckling R, Wright B: Improving access to psychological therapy: initial evaluation of two UK demonstration. sites. Behav Res Ther. 2009, 47: 910-920. 10.1016/j.brat.2009.07.010.View ArticlePubMedPubMed CentralGoogle Scholar
- Richards DA, Suckling R: Improving access to psychological therapies: phase IV prospective cohort study. Br J Clin Psychol. 2009, 48: 377-396. 10.1348/014466509X405178.View ArticlePubMedGoogle Scholar
- Bowman D, Scogin F, Floyd M, Patton E, Gist L: Efficacy of self-examination therapy in the treatment of generalized anxiety disorder. J Couns Psychol. 1995, 44: 267-273.View ArticleGoogle Scholar
- Bijl RV, Ravelli A, van Zessen G: Prevalence of psychiatric disorder in the general population: results of the Netherlands Mental Health Survey and Incidence Study (NEMESIS). Soc Psychiatr Epidemiology. 1998, 33: 587-595. 10.1007/s001270050098.View ArticleGoogle Scholar
- van SA, Seekles W, van ’t Veer-Tazelaar NJ, Beekman AT, Cuijpers P: Stepped care for depression in primary care: what should be offered and how?. Med J Aust. 2010, 192: S36-S39.Google Scholar
- Beck AT, Steer RA, Brown GK: Manual for the Beck Depression Inventory-II. 1996, San Antonio, TX: Psychological CorporationGoogle Scholar
- Spinhoven P, Ormel J, Sloekers PPA, Kempen G, Speckens AEM, van Hemert AM: A validation study of the Hospital Anxiety and Depression Scale (HADS) in different groups of Dutch subjects. Psychol Med. 1997, 27: 363-370. 10.1017/S0033291796004382.View ArticlePubMedGoogle Scholar
- Bakker AB, Demerouti E, Schaufeli WB: Validation of the Maslach burnout inventory – general survey: an internet study. Anxiety Stress Coping. 2002, 15: 245-260. 10.1080/1061580021000020716.View ArticleGoogle Scholar
- Demerouti E, Bakker AB, Vardakou I, Kantas A: The convergent validity of two burnout instruments: a multitrait-multimethod analysis. Eur J Psychol Assess. 2003, 18: 296-307.Google Scholar
- Brooks R: EuroQol: the current state of play. Health Policy. 1996, 37: 53-72. 10.1016/0168-8510(96)00822-6.View ArticlePubMedGoogle Scholar
- Cohen J: Statistical power analysis for the behavioral sciences. 1988, Hillsdale, NJ: Erlbaum, 2Google Scholar
- Jacobsen , Truax : Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991, 59 (1): 12-19.View ArticleGoogle Scholar
- Llardi SS, Craighead WE: The role of nonspecific factors in cognitive-behavior therapy for depression. Clin Psychol Sci Pract. 1994, 1: 138-155. 10.1111/j.1468-2850.1994.tb00016.x.View ArticleGoogle Scholar
- Cuijpers P, van SA, Warmerdam L: Problem solving therapies for depression: a meta-analysis. Eur Psychiatry. 2007, 22: 9-15. 10.1016/j.eurpsy.2006.11.001.View ArticlePubMedGoogle Scholar
- Ström L, Pettersson R, Andersson G: Internet-based treatment for insomnia: a controlled evaluation. J Consult Clin Psychol. 2004, 72: 113-120.View ArticlePubMedGoogle Scholar
- Eysenbach G: The law of attrition. J Med Internet Res. 2005, 7: e2-10.2196/jmir.7.1.e2.View ArticleGoogle Scholar
- Carlbring P, Westling BE, Ljungstrand P, Ekselius L, Andersoon G: Treatment of panic disorder via the internet: a randomized clinical trial of a self-help program. Beh Ther. 2001, 32: 751-764. 10.1016/S0005-7894(01)80019-8.View ArticleGoogle Scholar
- Kazantzis N, Deane FP, Ronan KR: Homework assignments in cognitive and behavioral therapy: a meta-analysis. Clinical Psychology. 2000, 7: 189-202.Google Scholar
- The pre-publication history for this paper can be accessed here:http://0-www.biomedcentral.com.brum.beds.ac.uk/1471-244X/13/43/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.