Going digital: a narrative overview of the clinical and organisational impacts of eHealth technologies in hospital practice

Considerable financial investment is currently being devoted in many countries to implementing potentially transformative eHealth technologies. For example, England has invested at least £12.8 billion in a National Program for Information Technology (NPfIT) for the National Health Service¹ and the Obama administration in the US has similarly committed to a US$34 billion eHealth investment in health care.^2,3 The eHealth market worldwide is predicted to reach US$308 billion by 2022.⁴ Much of this expenditure has been directed towards digitising hospital practice. Queensland Health has embarked on an A$1.3 billion state-wide campaign to digitise all its major public hospitals by 2020.⁵ Such large-scale expenditure has been justified on the grounds that electronic medical records (EMRs), electronic prescribing (ePrescribing) and associated computerised provider (or physician) order entry systems (CPOE) and computerised decision support systems (CDSS) will help address the problems of variable quality and safety in modern health care, improve efficiency and contain rising healthcare costs.^5–9 However, the evidence base underpinning such claims remains uncertain.

The aim of the present study was to generate a narrative overview of systematic reviews and reviews of reviews of contemporary eHealth technologies that have assessed their effects on the quality, safety and resource utilisation of hospital-based health care delivery compared with traditional paper-based systems of care.

Given the wide spectrum of eHealth technologies, the foci of this review were those having two functions (with some inevitable overlap): (1) primary technology to enable the storage, retrieval and transmission of clinical data (EMRs); and (2) auxiliary technologies to support clinical decision making (CPOE, ePrescribing and CDSS). Technologies supporting remote care (Telehealth, Telemonitoring and similar) were excluded from the analysis, as were patient-focused interventions (patient-controlled EMRs, electronic messaging or education). Given the rapid evolution of eHealth technologies, we restricted the present review to reports published in the past 5 years to ensure relevance to current deliberations.

Established Cochrane-based systematic review principles were used to search for relevant systematic reviews and reviews of such reviews. We used the pre-specified PubMed Clinical Queries search string for ‘electronic health record’ (see Item S1, available as Supplementary Material to this paper) and ‘article type = review’ and ‘publication date = last 5 years’ filters to search PubMed, MEDLINE and Cochrane Library contents for reviews published in English-language journals, either in print or online, from 1 January 2010 to 31 October 2015. Additional searches were performed using PubMed Clinical Queries function (with ‘systematic reviews’ search filter) and relevant search terms for CPOE, ePrescribing and CDSS. The bibliographies of retrieved reviews were scrutinised to find additional reviews. Articles were screened for inclusion on the basis of: (1) reference to the study as being a systematic review or a review of systematic reviews by the authors within the title, abstract or text; and/or (2) evidence from the description of the methods that systematic review principles had been used in searching and appraising the evidence. Reviews that contained randomised trials and/or observational studies were included in the analysis. Studies were excluded if, for any of the four eHealth technologies: (1) they focused only on single disciplines (e.g. oncology, mental health), single class of investigations or medications or single vendor systems; (2) dealt primarily with implementation issues; or (3) were conducted exclusively or predominantly in non-hospital settings or in developing countries. A complete list of inclusion and exclusion criteria is provided in Item S2. All potentially suitable reviews were independently assessed for inclusion by two reviewers (JK and IAS) and agreement reached by consensus. A log of excluded studies and reasons for exclusion is available on request from the authors.

The methodological quality and risk of bias of each review was critically appraised using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR; https://amstar.ca/Amstar_Checklist.php, verified 17 September 2015) tool for systematic reviews (Item S3). Reviews were rated as high quality on the basis of AMSTAR scores ≥8.

Data from included reviews were abstracted using a standardised format by one reviewer (JK) and cross-checked by a second reviewer (IAS). In synthesising the data, the heterogeneity of reviews was too diverse in terms of eTechnologies and outcome measures to allow meta-analysis of pooled data and therefore a narrative synthesis was undertaken.

The outcome measures, based on a typology of eHealth technology functions and effects developed by Black et al.,⁶ are listed below.

1. For EMR alone, the benefits (compared with paper-based systems) were categorised as data security, legibility, accessibility, completeness, comprehensiveness, efficiency and secondary uses. The harms were categorised as paper persistence, patient disengagement, insecure data, increased time for data entry and increased costs.

2. For CPOE, the benefits were categorised as resource utilisation, indicated care, patient outcomes (all-cause mortality, disease-specific mortality, disease-specific events, symptom measures, quality of life scores (as stated by the authors)), cost savings and time savings. The harms were categorised as increased time to perform task, interruptions, increased costs and workarounds. Where reviews reported findings in the CPOE category that were related to orders for medications, we thought it more appropriate to place these in the ePrescribing category for analysis.

3. For ePrescribing, the benefits were categorised as surrogate outcomes (as stated by the authors), guideline adherence, safer prescribing, communication, patient outcomes (as listed above), resource and/or cost savings and time savings. The harms were categorised as patient harm (adverse drug-related events (ADEs) or mortality), increased time to perform task and increased costs.

4. For CDSS, the benefits were categorised as indicated care, guideline adherence, surrogate outcomes (as stated by the authors) and patient outcomes (as stated above). The harms were categorised as practitioner performance and patient outcomes. Although we anticipated many CDSS would be applied to CPOE and ePrescribing functions, we elected to retain them in the CDSS category in keeping with their primary intended function.

Estimates of effect related to outcome measures were defined as: (1) qualitative measures, as stated by the authors (i.e. no, small (modest), moderate or large effect); (2) standardised effect size (ES), where an ES of 0.2 indicates a small effect, an ES of 0.5 indicates a moderate effect and an ES of 0.8 indicates a large effect; (3) relative risk (RR) or odds ratio (OR) for benefit (or harm); and (4) absolute risk reduction (ARR) or increase (ARI) for benefit or harm respectively.

Where review authors assessed the quality of the primary studies using numerical scoring methods (expressed as mean quality scores for all studies) or rated the quality of evidence as high, moderate, low, or very low based on the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) system,¹⁰ these scores or ratings were reported.

The search strategy retrieved 1197 abstracts for screening, of which 1139 were rejected, leaving 58 articles that were subject to full-text review. Of these, 21 articles were included in the overview (see Fig. 1), comprising 19 systematic reviews^{7–9,11–26} and two reviews of systematic reviews.^6,27 In all, 18 of the 21 (85.7%) articles evaluated CDSS, ^{6–9,11–19,22–24,26,27} five (23.8%) evaluated data information retrieval and transfer,^6,9,20,25,27 six (28.6%) measured effects of CPOE^{6,9,16,21,22,27} and three (14.3%) assessed ePrescribing.^6,16,25 Most articles evaluated more than one eHealth technology and nine of the 21 (42.9%) articles were rated as high quality.^{6,8,9,15,18,19,20,21,24} A summary of evidence from the studies included and interpretive comments made by their authors are given in Appendix 1 for systematic reviews and in Appendix 2 for reviews of reviews, listed in descending level of quality (and in chronological order within each level); key findings from each type of review are presented below.

Fig. 1.  Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of study selection. EMR, electronic medical records.

One low-quality review reported enhanced communication of information between providers and a 22% reduction in documentation time.²⁵ In a high-quality review, there was moderate-quality evidence of a 15% reduction in hospitalisations and small decreases in both the length of hospital stay and the number of patient visits to emergency departments as a result of electronically generated reports of investigations containing care recommendations.²⁰ The same review noted no improvement in timeliness of discharge summaries to primary care providers and no effects on disease-specific processes of care or clinical outcomes.²⁰ Two reviews, one low quality²³ and another high quality,⁹ that examined in-hospital mortality showed no effects from EMR. However, EMRs that included electronic surveillance systems for monitoring patient deterioration were associated with a 15% reduction in in-hospital mortality.⁹

In one high-quality review, there was limited evidence of improvement in data security, interpretability, accessibility, comprehensiveness and search and retrieval functions for secondary uses such as research.⁶ One low-quality review noted modest effects of EMRs on information accuracy and completeness, with just over half the included studies reporting positive metrics for data storage, retrieval and transfer.²⁷ There were also trends towards improved organisational efficiency and greater use of health information exchange technology.^6,27 Although there were varying levels of sophistication and functionality across home-developed and commercial systems, time efficiency depended on the skills of end-users.⁶

One high-quality review reported no effects of CPOE on hospital mortality or length of stay.⁹

In one low-quality review, positive feedback was reported for CPOE in two-thirds of 287 studies, but there was no effect on clinical performance or overall costs.²⁷ In a high-quality review of 53 reviews, modest effects were seen on turnaround times for processing and delivering laboratory investigation, as well as on practitioner performance.⁶ The same review noted small time and cost savings overall, but there was also evidence of increased time to interact with CPOE systems in order to address alerts and override or confirm orders, leading to interruptions and workarounds at the point of care.⁶

One high-quality review quantified the benefits of ePrescribing as a 54% decrease in medication errors and a 53% decrease in ADEs,²¹ whereas the corresponding reductions seen in a low-quality review were 54% and 34% respectively.²⁵ However, a twofold risk of new prescribing errors (e.g. concurrent submission of duplicate orders) was reported in two of 16 studies, which could potentially be disruptive to work flows but, to date, has not been associated with patient harm.²¹

One high-quality review reported moderate evidence of improvement in organisational efficiency, more accurate communication between prescribers and pharmacists, and limited evidence of decreased turnaround times in supplying stock to wards and filling discharge prescriptions.⁶ The same review reported limited to moderate evidence of reductions in medication errors and ADEs, increases in more appropriate prescribing, greater guideline adherence, improvement in disease outcomes and savings on cost and resource use.⁶ However, these benefits came at the cost of introducing alert fatigue and inadvertent selection of incorrect type and dosage of medications.⁶

Multiple CDSS were described with variable levels of sophistication with regard to inputs, targeted goals and prompts and inference mechanisms. High-quality reviews showed moderate evidence (more than 50% of primary studies were positive) of improvements in processes of care¹⁵ and appropriateness of test ordering.²⁴ In one review there was moderate-quality evidence of a 42% increase in preventive care services and a 72% increase in ordering or completion of recommended clinical investigations, and high-quality evidence of a 57% increase in adherence to treatment guidelines.¹⁷ However, the same review reported no effects on mortality, ADEs, length of stay or clinician confidence in patient care.¹⁷ In contrast, in another high-quality review, morbid events were reduced by 18% across a range of clinical conditions,⁸ whereas in a third high-quality review, in-hospital mortality was reduced by 17%, which was just below statistical significance (P = 0.05).⁹ One low-quality review reported a twofold increase in adherence to clinical recommendations, a 42% increase in the use of preventive care services and a 57% increase in the appropriate use of medical treatments, in association with a 12% decrease in the incidence of morbid events.²² However, this same review noted that, in the presence of alert fatigue, adherence to advice decreased by 63%.²² Another low-quality review noted moderate effects in reducing inappropriate diagnostic imaging and modest reductions in overall use, but also slight increase (~7%) in failure to order appropriate tests when indicated.²³ Other low-quality reviews noted evidence of less ordering of redundant tests^7,11,26 (up to 18% less¹¹), increased use of alerts and reminders^12,13 and diagnostic and medication dosing aids,¹³ improved processes of care,^12,14 lower costs,¹⁴ a 33% greater adherence to guidelines²⁵ and a 60% decrease in turnaround times.²⁶ With regard to chronic disease management, one low-quality review reported improved management of diabetes only,¹² whereas in another review, management of other chronic diseases, such as ischaemic heart disease,¹⁸ hypertension and chronic obstructive pulmonary disease, also benefited.¹³ Various reviews showed more significant optimisation of the use of high-risk drugs in certain patient subgroups, such as those receiving insulin,¹⁶ vitamin K antagonists,¹⁵ antibiotics and anti-rejection drugs.¹⁹

One high-quality review noted evidence of improved delivery of indicated care⁶ and guideline adherence,⁶ whereas a low-quality review²⁷ noted increased use of alerts and reminders.

The present narrative overview provides an update on the cumulative evidence of clinical and organisational effects of contemporary eHealth technologies in hospital practice. The current evidence base is limited for EMR simply as a data storage, retrieval and transfer platform, and for CPOE. In contrast, ePrescribing and CDSS have attracted greater research interest, probably because these systems have the greatest potential to directly and significantly affect patient care and outcomes. However, the overall quality of evidence is low, with only nine of 21 reviews being rated as high quality. Moreover, the results of different reviews, even those ostensibly studying the same question, yield somewhat inconsistent results, which may reflect, in part, the heterogeneity of the populations studied, technologies analysed and outcomes measured. Positive effects in many reviews are often small to moderate in magnitude and based on low- to moderate-quality evidence. There is very limited and conflicting evidence of the effects of eHealth technologies on patient-important clinical outcomes, such as morbid events, mortality and unplanned readmissions. Measures of hospital bed use, equity of access, resource utilisation, patient satisfaction or quality of life measures, or provider satisfaction and perceived ease of use, were rarely, if ever, reported. Although some reviews made mention of overall costs, data relating to cost-effectiveness remains sparse and is no doubt explained by the complexity of measuring direct and indirect costs of development and implementation of eHealth technologies over a period of time sufficiently long enough to allow adequate evaluation.

The present overview provides a current synopsis of the evidence of effect of currently available eHealth technologies that may assist project groups in developing business cases for local design and implementation. Strengths include a comprehensive search for systematic reviews and reviews of reviews from the literature and an assessment of their quality using a validated appraisal instrument. Detailed information on study characteristics, process and outcome measures according to four categories of technology functions, and interpretive comments, were extracted from each review, enabling readers to better assess relevant intervention effects.

Limitations relate to inadequate indexing of eHealth technologies in the literature, although the search method was one developed and endorsed by PubMed and it is unlikely that sentinel articles were missed. The scope of the present review was restricted to articles published in the past 5 years, which may have led to oversight of effective technologies that have withstood the test of time, although many reviews included individual studies dating back more than two decades. We did not search grey literature from websites or other sources given the potential for bias in reports not subject to peer review. We only assessed clinical and organisational effects devoid of any consideration of contextual issues around implementation, which may have affected the results. We concede overviews, by aggregating results from multiple primary studies, can only generate estimates of ‘average’ effects that may totally obscure highly positive results in individual trials. In addition, reviews rarely distinguished between studies that evaluated eHealth technologies as before–after ‘brownfield’ designs (i.e. an existing hospital converting from paper-based to computerised systems) or comparative ‘greenfield’ designs (newly built computerised hospitals or units compared with concurrent or historical controls). However, our aim was to generate effect estimates that were broadly generalisable.

Our categorisation of results of some reviews into CPOE, ePrescribing or CDSS could be regarded as arbitrary, but overlap of these functions was, at times, unavoidable and our choice of category was explained. There is the also the potential for overemphasising the findings of particular reviews (and their included primary studies) by including multiple reviews and reviews of reviews that duplicate the same studies multiple times. However, adopting this approach affords the opportunity for noting consistency in, and hence potential robustness of, the findings of different reviews performed by different authors. Although we were unable to assess the quality of individual trials in every review, we did cite the methods and results used to conduct such assessments whenever reported by review authors, and the AMSTAR quality criteria we used to appraise the reviews take account of whether such assessments had been undertaken. We were unable to pool data across all reviews, but reported quantitative results for those reviews that undertook meta-analyses. Many evaluations were conducted in large academic institutions by potentially conflicted developers of the eHealth technologies, which raises the potential for information and publication bias leading to overestimation of benefits, with most reviews making no mention of potential harms. Where review authors tested for such bias, we have reported their findings.

The authors have been involved in the recent digitisation of a large tertiary adult hospital in Brisbane that incorporates all four eHealth functions mentioned above. Presenting a synthesis of evidence of effects helped inform clinicians’ and managers’ views of the benefits (and harms) of digitisation, and assisted in gaining hospital-wide acceptance for such a major transformational change.

This overview suggests EMRs have the potential to increase the accuracy and completeness of clinical information and to reduce documentation time. Improvements in information transfer and organisational efficiency may translate into reduced hospitalisations, emergency visits and redundant test requests. Whether EMRs in the absence of auxiliary support systems affect clinical outcomes or overall costs remains unclear. Similarly, although CPOE appears to improve laboratory turnaround times, this is offset by increased interaction times, workflow interruptions and workarounds. Neither EMR nor CPOE has been shown to reduce mortality, length of hospital stay or overall costs.

There is reasonably strong evidence that ePrescribing significantly reduces medication errors and ADEs and may further increase patient safety and organisational efficiency with reminders relating to significant drug interactions that may otherwise be missed. Eliminating illegible handwriting and inappropriate dose prescribing by physicians, and improving communication with pharmacists, are proven benefits. However, there is potential for alert fatigue and duplication or wrong selection of medication type and dosage.

Evidence for improving processes and quality of care that affect clinical outcomes is currently strongest for CDSS, especially those used at the point of care and integrated well into workflows. Reviews have shown that such systems substantially increase clinician adherence to guidelines, appropriateness of disease and treatment monitoring and optimal use of medications. However, customisation of CDSS applications to the context of patient-centred care and management of highly complex patients with multimorbidity remains an ongoing challenge.

This overview of recently published systematic reviews of eHealth technologies may help inform decisions to invest in eHealth technologies in hospitals throughout Australia. Overall, eHealth technologies, especially ePrescribing and CDSS, appear effective in improving health care processes and outcomes across diverse settings using both commercially and locally developed systems.

eHealth technologies are evolving rapidly and much is still to be learned as to how these tools should be designed and used in ways that optimise their effectiveness. Evaluation methodologies most suited to assessing benefits and harms of these technologies, in both quantitative and qualitative terms, need to be deployed and refined over time. More rigorous field research targeting hospitals undergoing digital transformation, and performed by independent, multidisciplinary research groups, is required to narrow the gap between theorised potential benefits of eHealth technologies and empirically demonstrated real-world improvements in patient care and outcomes and efficient use of resources.

None declared.