Controlled clinical trial of a self-management program for people with mental illness in an adult mental health service – the Optimal Health Program (OHP)

Self-management is a patient-focussed approach to managing chronic conditions such as diabetes, asthma and cardiac disease^1–4 that challenges traditional models of health provision by encouraging people to be actively involved in their own healthcare.⁵ The common elements of self-management have been well described^2,3,5 and include: active participation of the individual in their treatment to minimise the effect of their condition, goal setting, problem solving and self-management support (patient, healthcare practitioner and healthcare system interventions designed to increase self-management behaviour).

Although ample evidence supports the efficacy of structured self-management programs for chronic conditions such as diabetes and asthma,⁶ far less research has evaluated this approach in the area of mental health, where conditions such as schizophrenia, psychosis and bipolar require on-going treatment.⁷ Although there is evidence that self-management programs improve outcomes for people with mental illness, services remain limited in their uptake.⁸ In many organisations, self-management programs continue to be offered as an additional activity, rather than as a core part of service delivery.⁹

A possible reason for this is that programs are often evaluated in research settings, and not in clinical settings, where routine service delivery is incorporated into the study. In highlighting the difficulties associated with translating evidence into practice, Gilbert et al.¹⁰ suggested some strategies for validating treatment programs that are currently used in service settings. For example, in Australian mental health services, routine outcome measurement has been introduced and is valuable in informing questions of service quality and effectiveness¹¹.

We report here outcomes of a study undertaken in a naturalistic setting which examined the effectiveness of a structured self-management program for people with a mental illness. We hypothesised that, compared with routine care, participation in the self-management program would improve health and social functioning and reduce hospitalisations. A secondary hypothesis was that the delivery of the program would be cost-effective.

This study was a quasi-experimental clinical trial, with a non-randomised control group. The treatment condition for the clinical trial was a psychosocial self-management program delivered as part of routine care within a community mental health service. The comparator was routine care alone (control condition).

The study was conducted in the Australian Capital Territory across four adult community mental health teams between 2003 and 2007. This project was approved by the ACT Health, Human Research and Ethics Committee.

The uniform organisation of mental health services within the ACT provided a clear advantage for conducting a study in a naturalistic setting. The ACT (population 330 000) is a single health jurisdiction with uniform mental health service delivery models across four geographically distinct areas. Public mental health services are provided by Mental Health ACT in a comprehensive and integrated service. Mental Health ACT has four adult (18–65 years) community mental health teams that provide services to residents within their designated region. They all offer clinical management, specialist assessment and treatment delivered by mental health professionals from multidisciplinary backgrounds. The majority of people treated by the adult community mental health services have a low prevalence disorder such as schizophrenia or bipolar disorder.

Routine outcome measures, including Health of the Nation Outcome Scale (HoNOS), are collected by all teams according to National Outcomes and Casemix Collection (NOCC) protocols and entered into a common database known as Mental Health Assessment Generation and Information Collection (MHAGIC).

Of the four adult mental health teams, two teams were allocated to the treatment condition and the other two to the control condition. The allocation of teams to the treatment or control condition was out of the control of the researchers. Teams were allocated according to the administrative requirements of Mental Health ACT rather than using a randomisation procedure such as formal cluster randomisation. Also, in this context individual randomisation of subjects was not possible (and would have raised issues regarding contamination if attempted).

The control condition consisted of routine mental healthcare, including medical treatment delivered using a case management approach.

The treatment condition consisted of routine care plus the Optimal Health Program (OHP). The OHP (Table 1) aims to provide education and skills to enable participants to manage their mental health in collaboration with services, carers and others. It consists of nine sessions that are delivered sequentially according to a structured treatment manual. Importantly, it can be delivered with groups or individual consumers by a range of health professionals, including nursing and allied health staff, and in both inpatient and community settings. When delivered within a service, it provides clinicians with a consistent approach and common language for supporting the development of self-management skills.

Table 1.  The OHP intervention

The OHP adopts a person-centred approach and a focus on health as defined by the consumers. The program is generic in that it is designed to address both the physical and psychosocial dimensions of the health of people with mental illness, who have very high rates of medical co-morbidity and premature mortality. It uses the ICANDO™ model, an applied version of the stress-vulnerability model, to develop consumers’ knowledge and self-efficacy about managing health and illness. Key aspects of the program include: coping strategies e.g. monitoring, goal setting; identifying social supports; and developing plans to (i) cope with daily stressors; (ii) manage early warning signs; and (iii) respond to an episode of illness. Consumers are provided with resources including a participant workbook and health journal to apply and reinforce information and skills covered during the sessions.

Case managers were trained to facilitate the OHP during a two-day workshop delivered by the first author and research clinicians. The workshop covered: (i) the theoretical concepts of the OHP; (ii) how to engage consumers and facilitate sessions; and (iii) the implementation process in the service. Following the workshop, two researchers provided supervision to clinicians in groups or individually to support delivery of the OHP and promote fidelity.

In this study, the OHP was offered to consumers of the teams allocated to the treatment condition. Consumers were provided with information about the program and if they gave consent to participate, they began individual sessions with their case-manager or were referred to the group program. Decisions about individual sessions or the group program were undertaken by the case-manager or the consumer’s preference.

The primary outcome for this study was health and social functioning measured on the HoNOS,¹² a 12-item, clinician-rated measure of treatment outcome. Each HoNOS item is rated from 0 (no problem) to 4 (very severe problem) and summed to give a total score. It is completed by case managers on admission, discharge, transfer of care and at 90-day intervals while a person is a consumer of mental health services.

As a baseline, the project coordinator ensured that consumers in the treatment condition had a HoNOS completed within the 3 months before commencing the OHP. All treatment consumers were observed over a maximum of 12 months after commencing OHP.

The secondary outcome, hospitalisation, was measured using hospital admission and discharge data. In the treatment group, data were collected during the 12 months before and 12 months after commencing OHP.

Control consumers were observed for an equivalent time frame for both HoNOS and hospitalisation data.

Data on these outcomes are routinely collected by Mental Health ACT and are able to be accessed using the MHAGIC database. This avoided bias and an additional burden on case managers. Additional data were collected by the project coordinator, including OHP start date, mode of delivery (group or individual) and number of sessions completed.

All consumers aged between 18 and 65 years old, with a diagnosed mental illness who were receiving treatment from the adult mental health service, were eligible to participate in this study. Consumers in the treatment sites were included in the analyses if they had an OHP start date recorded. Consumers were excluded if they did not have a baseline HoNOS or they did not have at least one post-baseline HoNOS.

We aimed to sample the control and treatment data in a way that would make the two groups comparable. Although HoNOS measurements were taken nominally every 3 months, the timing of HoNOS data was not evenly spaced and the frequency of measurements was variable across consumers. In addition, some consumers only had data for part of the periods of observation.

All analyses were performed using Stata 10.1.¹³

All consumers with at least one post-baseline HoNOS measurement were included in the initial analysis. After sampling, there remained baseline differences in the means and standard deviations of relevant variables between the treatment and control groups. Consequently, a propensity score¹⁴ was estimated to facilitate analysis using those consumers within the common support, so that the consumers in the treatment group were comparable with those in the control group based on the baseline covariates (listed in the first column of Table 2), thereby improving the balance of the treatment and control groups at baseline.

Table 2.  Comparison of treatment (n = 44) and control (n = 196) samples after applying propensity score common support

Because not all of the baseline differences between treatment and control could be eliminated simultaneously with the propensity score analysis, change scores (from before to after) were used to give a fair comparison. The change associated with receiving OHP was assessed by calculating the means and standard errors of the HoNOS change scores for the treatment and control groups, using the Stata survey (svy) commands to account for the multiple (dependent) post-baseline HoNOS measurements. In addition, the change in HoNOS was examined using linear regression, looking at stratification on quintiles of the propensity score.

The changes in the percentage of time spent in hospital and the rate of admissions, 12 months before and 12 months after, for treatment and control, were analysed using permutation tests and a bootstrap CI¹⁵ for the difference was calculated.

A prospective economic analysis was also undertaken. This analysis incorporated hospitalisation costs obtained from Mental Health ACT ($800 per patient per day) and costs of the intervention ($240 per consumer). The intervention costs were estimated using the following assumptions: (i) set-up costs, which included facilitator training and preparation time, and (ii) delivery costs, which incorporated delivery mode (29% individual, single facilitator; 71% group, two facilitators, six consumers), clinician salary and consumers’ materials. These data were translated into a net reduction in yearly cost (saving) over ‘before’ and ‘after’ time periods for each consumer. For the treatment participants, the calculation incorporated OHP intervention and hospitalisation costs. For the control consumers, only hospitalisation costs were used.

Comparing the before and after cost differences for treatment and control gives the average net saving associated with delivering the OHP. Bootstrap standard errors¹⁵ were calculated for the net saving.

All data were collected between 2003 and 2007. Consumers at the treatment sites were recruited from April 2004 to July 2006: those who participated in the treatment were observed for 12 months after commencing OHP. Consumers at the control sites were observed between March 2003 and October 2005.

A flow diagram for study consumers is given in Fig. 1. In the treatment group, 120 consumers were offered the OHP by their case manager. Of the 92 consumers starting OHP, 27 received the intervention one-to-one and 65 received the intervention in group format. Of those who received a group intervention, the mean group size was 5.9 with standard deviation 2.5. The mean number of sessions completed was 6.3 with standard deviation 2.5 (10 consumers did not have the number of sessions recorded). There was also attrition due to some consumers having insufficient data or being outside the propensity score common support.

Fig. 1.  Participant flow.

The baseline characteristics of the treatment and control groups, after applying propensity common support to balance the characteristics of the two groups, are compared in Table 2.

The mean change in HoNOS score between baseline and post-baseline for the treatment group was –2.11 (95% CI –3.03 to –1.19) and for the control group, the mean change was 1.06 (95%CI 0.11 to 2.01), where a negative change means a reduction (improvement) in HoNOS score. The difference between treatment and control groups was –3.17 (95% CI –4.49 to –1.84; P < 0.001), showing that, on average, the treatment group improved significantly in HoNOS scores compared to the control group. To place these results in context and facilitate comparison with the results in the literature,¹¹ we express these results in terms of effect sizes based on a standard deviation of 6.67. The average change for the treatment group had effect size 0.32 indicating improvement (95% CI 0.18 to 0.45) and the change for the control group had effect size –0.16 indicating slight deterioration (95% CI –0.30 to –0.02). For the difference between treatment and control, the effect size was 0.47 (95% CI 0.28 to 0.67).

Additional regression analysis showed that the results for the change in HoNOS were consistent across the five strata based on propensity score (Fig. 2). Overall, the reduction in HoNOS scores observed in the treatment group indicates a significant improvement in health and social functioning relative to the control group.

Fig. 2.  (a) Regression model predictions for change in HoNOS, treatment compared with control over the five propensity score strata, based on the centre of the time period. Negative values indicate a reduction in symptoms (improvement) and positive values indicate deterioration. (b) Box plots of baseline HoNOS scores over the five strata.

Analysis of hospitalisation data showed a reduction in hospital service usage in the treatment group relative to the control group (Table 3). However, although the permutation tests of the differences were not statistically significant for both percentage of time spent in hospital (P = 0.07) and hospital admission rates (P = 0.2), the corresponding bootstrap 95% confidence intervals were borderline significant (percentage of time in hospital: 0.34 to 4.41; number of admissions 0.04 to 1.00).

Table 3.  Percentage of time in hospital and admission, before and after baseline, and net cost saving

The economic analysis (Table 3), based on the percentage of time spent in hospital, suggested that there are significant potential savings associated with the OHP intervention.

The distribution of the change in cost from before to after for the treatment and control groups is illustrated in Fig. 3. OHP intervention was estimated to save in excess of $6000 (95% CI $744 to $12 656) per consumer per year.

Fig. 3.  Net reduction in yearly cost (i.e. cost in the period before baseline minus cost in the period after baseline) for participants in control and treatment groups. Positive values indicate a cost saving. Note the absence of a long negative tail for the treatment group.

This study shows promising results for incorporating a self-management program into routine care to improve the health and social functioning of mental health consumers. Added to this there appears to be significant cost savings to the health service in implementing such a program. The key strengths of this study are its controlled design and the naturalistic setting in which it was undertaken, which supports the generalisability of findings to other settings.

Our study builds on the work of Burgess and colleagues,¹¹ showing the usefulness of routinely collected data for evaluating service improvement initiatives, such as the implementation of a new program. The medium effect size of the difference between treatment and control change scores indicates moderate clinical significance.

Our study also supports the feasibility of delivering a self-management program within routine mental health services. The OHP used in this study is flexible in its mode of delivery and can be facilitated by a range of mental health professionals. It is also relatively brief compared with other programs. For example, the Illness Management and Recovery program¹⁶ is delivered over 9 months. The high uptake of the OHP and rates of session completion suggest that this program is acceptable to consumers. Longer term follow-up would provide information on the effectiveness of this program over time.

Several limitations to our study warrant mention.

Although not feasible, randomisation of individual subjects to the treatment or control group would have been ideal to eliminate confounding. That the baseline characteristics of the two groups differed highlighted this confounding issue. To account for this, we undertook propensity score analysis, which is an accepted method for adjusting for baseline differences between two (or more) groups in a comparative study.

The analysis used a population average method and care is required when generalising from the confidence intervals in the results. The results generalise in that, if the data collection were repeated using the same sampling scheme and reanalysed, then the estimated average difference would be within the relevant confidence interval 95% of the time.

Furthermore, the self selection of the program by the clients themselves may have introduced selection bias, given that not all those who were offered OHP started the program. Presumably those who selected not to do the OHP when offered were, on average, more unwell at baseline. This could have caused bias either way. If consumers who are unwell at baseline do not respond as well, then the OHP treatment could be overestimated. Alternatively, if consumers who were unwell at baseline, with initially higher HoNOS scores, were more likely to show improvement than those who started with an intermediate HoNOS score, then the treatment effect would be underestimated in this study.

In terms of outcomes, only the HoNOS total score was measured. Other routine outcome measures e.g. Life Skills Profile¹⁷ and BASIS-32¹⁸ could have been used also, but we chose the HoNOS as it was the most suitable measure to address the primary hypothesis and had the most data available. In addition, mental health service contacts other than hospitalisation are another source of data (not available in this study) that would give a more complete picture of mental health service use.

In conclusion, we believe this study contributes to the body of evidence supporting self-management as an adjunct to routine care in mental health services. It also shows how routine data collection by health services can be used in a meaningful way.

The authors declare that no conflicts of interest exist.