Are we doing it right? We need to evaluate the current approaches for implementation of digital health systems

Keywords: healthcare information technology, healthcare software, medical software, implementation, project management, digital health, digital transformation, health informatics, Agile project management, Traditional project management.

Complex healthcare information technology (HIT), particularly off-the-shelf commercial healthcare software solutions, are increasingly being deployed in hospitals and healthcare services worldwide.¹ The adoption of these systems is expensive, requires significant training efforts and often causes temporary loss of efficiency and reporting.^2,3 Furthermore, HIT projects are characterised by high complexity of content, need for integration of multiple data sources and data types (e.g. patient information, radiology tests and laboratory tests), high standards for privacy protection and safety, and the need to satisfy a diverse range of powerful stakeholders.^3,4 HIT projects are subject to mid-project budget and compliance requirements reviews, and often have to be designed with provisions for data management, auditing and interoperability. The projects are now often deployed into existing complex digital environments requiring interactive and complex clinical workflow integration. Not surprisingly, failure rates for HIT projects have historically been higher than in other industries,^5–7 and large HIT projects continue to experience cost and/or schedule overruns and, in some cases, terminations.^8–13

Project failure refers to the inability of a project to meet project goals in terms of scope, schedule and budget, and/or project cancellation.^3,6,7 A negative bias against publication of failed projects in the health care and specifically in the medical informatics literature makes it difficult to get accurate statistics about failed HIT projects across the globe.⁶ However, large HIT implementations that have been cancelled or experienced serious problems in meeting set budget, scope or schedule targets have been reported in the United Kingdom, United States, Canada, Australia, Norway, and the Netherlands.^{6,8,11,14–21} McLoughlin et al. have extensively studied implementation of large HIT in Organisation for Economic Co-operation and Development (OECD) countries and report that project failures have occurred even in Denmark, which is considered a world leader in HIT.²² HIT projects fail due to various technical, functional, information and user-related factors.^7,23 The majority of failures (up to 65%)⁷ have been reported to relate to inefficiencies in project management that include insufficient resources, lack of detailed project and strategic planning, inadequate managerial support and insufficient budgets.^7,23,24 Inefficient project management may be caused by various organisational factors that include enforced implementation, differences in views on processes, budget and policy restrictions.²³

Remarkably, despite this repeated failure of current project approaches, there is limited evidence for health services looking to utilise evidence-based project management techniques as they embark on complex digital health programs. Consequently, they often rely heavily on external consultants, contractors, and advisory companies for all project management decisions and, in some cases, without knowledge of whether the approaches employed by external project teams are evidence-based. Our aim in this article is to examine evidence underpinning current project management approaches for implementing commercial, off-the-shelf HIT solutions including complex healthcare software in large digital health programs. This will provide hospitals and health services with evidence-based decision support as they commission large, complex digital health programs.

To minimise the risk of failure, software and other information technology (IT) implementations are guided by project management techniques.^25–27 Project management approaches are broadly classified as ‘Traditional’ or ‘Agile’.^28,29 Traditional approaches involve an extensive collection of detailed user requirements followed by linear execution, with limited end-user involvement until near delivery. The Waterfall methodology, first formally described by Winston Royce in 1970,³⁰ is a classic traditional project management approach. Other traditional methodologies include PRINCE2 (an updated version of Projects in Controlled Environments, which was developed by the British government for IT projects)³¹ and PMBOK (project management body of knowledge) created by the Project Management Institute.³²

In contrast, Agile approaches are more iterative and consultative, with rapid cycles of execution involving end-user feedback.^33,34 Agile approaches emerged in the 1990s as alternatives to the traditional approaches that were plagued by time and/or cost overruns, and inability to adapt to changing specifications.^35,36 Agile is an umbrella term for a variety of methodologies that include Scrum, eXtreme programming (XP), dynamic systems development method (DSDM), Crystal, unified development process (UDP) and feature-driven development (FDD).^36,37 Traditional and Agile approaches for project management can also be applied concurrently in a hybrid approach.³⁸

In Australia, some government- or organisation-level policies prescribe the use of Traditional or Agile project management approaches for managing HIT implementations.^39–42 In some cases, blanket use of traditional approaches is mandated for all IT projects (including HIT), albeit with little evidence underpinning their selection.^40,42 The recent failures of several high-profile HIT projects^8–13 should prompt reconsideration of traditional project management approaches for implementing such projects, particularly when large off-the-shelf, complex software are implemented into existing digital ecosystems in hospitals or across health systems. We define large complex healthcare software as vendor-supplied (off-the-shelf) applications requiring major digital transformation for healthcare organisations (e.g. electronic medical records, clinical and health information management systems, scheduling and billing programs).

The healthcare sector across Australia is currently spending billions of dollars on implementing complex HIT projects.⁴³ The spending is projected to grow in 2021 and beyond as healthcare organisations continue digital transformation efforts and invest in technologies that support data-driven decision-making.⁴⁴ The socioeconomic and political stakes in these projects are high and steps must be taken to ensure successful implementations. Adopting a suitable project management approach is a major factor for achieving success^11,24 because managing a project using an unsuitable methodology can severely damage the chances of success.^28,45,46

Theoretically, Traditional or Agile approaches may be employed for implementing HIT projects,²⁶ but there are suggestions that traditional approaches may not be suitable for this purpose as they fail to take into account existing complex digital ecosystems and varying clinical workflows in hospitals or health services.^20,34,47 Traditional approaches were originally designed for linear engineering projects, assume that projects are predictable, the environment is ‘controlled’ and emphasise collection of requirements before execution;^33,34 however, HIT implementations involve a complex bi-directional interaction of technical development, often chaotic environment and workflow transformation. Also, detailed collection of clinical requirements for a local HIT implementation may be unnecessary for centrally purchased off-the-shelf commercial solutions, given the limited ability to configure such products beyond simple content specific to a new setting.^48–50

Agile approaches, which are more flexible than Traditional approaches, have been suggested for implementing HIT for large digital health programs.^51–53 Unlike Traditional approaches, Agile approaches value continuous adaptation to changing requirements, regular delivery of functioning software, frequent in-person communication between project teams and customers as well as within project teams, collaboration between project team and customers, and autonomous teams.³³ In an educational paper, Kitzmiller et al. suggest Agile approaches for several components of project management (e.g. communication techniques, team selection, risk assessment and evaluation) in a HIT implementation project. For example, instead of the project team comprising project managers and IT staff only, in an Agile approach, the team would additionally include end-users or instead of restricting communication to the project team, communication channels are open to end-users throughout the project.⁵³ Hybrid approaches that combine aspects of the Agile and Traditional project management approaches have been recommended by others as a way of speeding up project execution and delivering better solutions for customers;^38,52,53 however, suggestions for Agile and Hybrid approaches are not supported by objective evidence of their effectiveness for implementing HIT projects. Although some project teams have been reported to perceive Agile or hybrid approaches as more suitable for managing HIT projects,⁵¹ objective evidence is needed to prove that Agile and hybrid approaches are indeed more effective at enabling HIT projects to meet budget, time and scope goals. But there is a stark absence of studies in mainstream academic and grey literature that systematically evaluate the comparative effectiveness of project management approaches for large HIT implementation projects.^52,53 Advocates for Agile approaches for implementing HIT projects also cite the lack of robust, objective evidence showing that switching from Traditional to Agile approaches improves healthcare IT implementations when cautioning against viewing Agile approaches as miracle fixes for failing projects.^52,53

Some articles in both mainstream academic and grey literature provide guidance and opinions on the implementation of healthcare IT, particularly electronic medical records, without describing which specific project management methodologies they may be reflecting on.^27,54–56 They also lack empirical evidence to support their guiding principles and often only cite unvalidated ‘lessons learnt’ or respondents’ opinions from single-location implementations.^51–53 Health care is an evidence-based discipline, and this should apply to implementation of HIT projects. Hospitals are complex, adaptive systems; therefore, as we evolve from simple bespoke digital systems built to detailed clinical requirements for a limited set of clinicians to the more modern implementation of commercial, off-the-shelf, integrated systems spanning hospitals, we also need to mature and evolve our approach to evidence-based change and implementation methods. Continuing to universally apply project management approaches in HIT implementations without evidence of effectiveness is counterproductive and unlikely to reduce failure rates.

The lack of studies describing specific project management approaches for large HIT implementation projects is remarkable given the sensitive nature of this work, the high failure rates, the patient safety considerations and the large budgets of these projects. More research is needed to define evidence-based approaches for efficiently delivering complex HIT solutions that are fit for purpose for our clinicians and consumers. Healthcare teams and leaders need to understand the varying approaches to HIT implementations for large digital health programs and to make informed and evidence-based decisions on their preferred approaches to these sensitive, risky and expensive projects.

The authors declare no competing interests.