Evaluation of three population health capacity building projects delivered by videoconferencing in NSW

In 2005, the NSW Telehealth Initiative called for submissions for innovation projects through the Telehealth Innovation Fund, and for the first time sought applications for population health interventions. Three population health projects were subsequently successful in securing funding. These projects sought to build the capacity of the health workforce to provide population health interventions and were delivered through live videoconferencing sessions between April and August 2007. Videoconferencing is a technology that involves the transmission of images, voice and data between two or more sites using telecommunication channels; through the extensive Telehealth network hundreds of videoconference sites have been established across the eight area health services (AHSs) in New South Wales (NSW) for the delivery of clinical services and administrative and professional development activities.^1,2 A condition of funding was that videoconferencing as a mode of delivery was evaluated from the perspective of those who delivered the projects. This article describes the findings of the evaluation.

Fall injury is a major cause of preventable injury-related hospitalisation and loss of independence among people aged 65 years and over in NSW. The NSW Clinical Excellence Commission implemented the Falls Prevention Telehealth Project to assist hospitals in rural AHSs to implement the National Guidelines (Preventing falls and harm from falls in older people. Best practice guidelines for Australian hospitals and residential aged care facilities) developed by the Australian Council for Safety and Quality in Health Care. The project was an education program that sought to prepare health professionals to implement best practice to prevent falls in NSW hospitals and aged-care facilities. Three videoconference sessions each of 2 hours duration were conducted across eight Telehealth sites – two from each of the four rural AHSs. Between 12 and 20 people attended at each site, with approximately 105 participants per session. The sessions were interactive and were recorded; a copy of the DVD/CD was distributed to the participants.

The Centre for Chronic Disease Prevention and Health Advancement within the NSW Department of Health piloted the delivery of an accredited, competency-based training program in the treatment of nicotine dependence using live videoconferencing as the delivery method. The training addressed two units of competency in tobacco use and evidence-based treatment of nicotine dependence within the national Vocational Education and Training accredited Population Health Training Package. Accredited trainers delivered 10 modules, each of 2 hours duration, to 27 sites and a total of 309 participants.³

The NSW Refugee Health Service, located at the Sydney South West Area Health Service, used videoconferencing to deliver a series of lectures/information sessions on refugee health issues to clinicians and population health staff. Six lectures of 1 hour duration were provided on the topics of infectious disease, child health, mental health, the physical impact of torture, oral health and nutrition, and refugee women’s health. Eleven sites participated with between 30 and 70 participants for each session. In addition, a small number of networking sessions for refugee health nurses working in rural and remote areas were conducted.

To determine the utility of using live videoconferencing to deliver population health training and other potential applications, an external evaluation was commissioned by the Population Health Division, NSW Department of Health; this was undertaken in July and August 2007.

A process evaluation examined the following three areas across each project: their organisation and delivery (including what worked well, what difficulties were experienced and the workload involved); the utility of videoconferencing for the delivery of the sessions; and the potential for future application of the technology.

The evaluation sought information about using the medium of videoconferencing to provide learning sessions for professional development in the population health area. It did not assess the impact or quality of the individual projects; separate evaluations were undertaken by each project to explore this. A consultant was contracted to carry out the evaluation. Oversight of the project was provided by an Advisory Group who initially helped to refine the evaluation questions. To allow the consultant to become familiar with the three projects and the processes associated with delivering learning sessions through videoconferencing, relevant background documentation was reviewed, preliminary discussions were held with each project manager, and one session of the refugee health project was observed.

Qualitative semi-structured interviews were conducted with key informants. Key informants were determined as being workers who were directly involved in and/or responsible for the planning and delivery of the projects. Interviews were conducted with 12 people and included:

• the project coordinators and other staff from each project

• people who acted as site facilitators at the remote sites

• presenters of the sessions

• NSW Health Telehealth Initiative staff.

The views of participants in the sessions were not sought. The majority of interviews were conducted face-to-face, however those held with rural informants were undertaken by telephone. The interviews were guided by a written protocol that was provided to the informant prior to their interview. To minimise the potential for recall bias the evaluation was undertaken at the same time the projects were being delivered, and the report completed in September 2007.

The information gathered through the interviews was analysed and a summary of preliminary findings prepared. The Advisory Group reviewed the preliminary findings and reviewed a draft of the report before it was finalised.

The projects were complex. Each involved the live delivery of learning sessions to between eight and 11 remote sites located in rural locations across NSW, with each session drawing up to 105 participants (Table 1). Despite this complexity the projects were all delivered successfully through live videoconferencing.

Table 1.  Description of sites and number of participants for three population health capacity building projects delivered by videoconferencing in NSW in 2007

Successful organisation and delivery of a session required a designated project coordinator with project management skills at the central delivery location and local facilitators at each remote site. Both these roles were found to be necessary. Box 1 summarises the core tasks undertaken by each. In all cases, the workload required, especially in the planning and organisation stages, was underestimated.

Box 1.  Roles and responsibilities fulfilled by project coordinators and site facilitators for three separate capacity building projects in population health delivered through videoconferencing in NSW in 2007

The effectiveness of videoconferencing is reliant on the technology; overall, the majority of technical problems experienced were considered to be minor. However, when technical disruptions did occur it was both frustrating and stressful to those organising a session. There were varying levels of technical support available both at the host and remote sites and where there was little infrastructure support, the sessions were vulnerable to disruption.

The timely availability of supporting resources, such as course materials and PowerPoint presentations, was an important aspect of providing training through videoconferencing, and the information technology systems in some rural locations could not manage the transfer of these large electronic files. Further, some participants did not have computer access to allow them to receive materials electronically.

All informants were satisfied that videoconferencing had met the needs of their projects and all considered it an appropriate technology through which to deliver population health learning initiatives.

The two main benefits highlighted were: the ability to deliver training to large numbers of people across multiple locations within a relatively short time and for reasonable costs; and the ability to improve access to high quality professional development for rural and remote workers. It was considered unlikely that initiatives of similar scope and reach could be delivered in a face-to-face mode due to the associated costs.

The informants identified many other benefits of videoconferencing. Both the smoking cessation and falls prevention projects were modules consisting of a number of short sessions designed for staged learning over time. These modules allowed for the assimilation and application of learning between sessions and for participants to discuss their experience in subsequent sessions with their peers.

The centralised delivery of learning sessions was associated with a number of perceived benefits. It allowed learning to be delivered in a consistent manner for all participants. The informants commented that other ways of rolling out training, such as train-the-trainer modes, can be less consistent in their delivery. It also allowed the participants direct access to experts and the opportunity to interact with them.

The development of learning communities was facilitated in a number of ways. Firstly, as it took less time to attend these sessions it encouraged clinicians to participate, allowing for multidisciplinary learning. Secondly, it expanded the networks of the participants and led to collaboration. These connections occurred between: participants attending at a site, participants at different sites, and participants and the central or statewide service that delivered the training. Some groups continued to meet after the sessions were completed.

Videoconferencing was seen as a valuable tool for building workforce capacity in population health and many ideas were offered about its future use. These included: providing access services (i.e. interpreter services) in remote areas; forming part of communication strategies for guidelines; and enabling collaboration between isolated practitioners, such as rural refugee health nurses.

Informants expressed caution about the infrastructure required to ensure the reliability of videoconference delivery. In addition, many projects use non-Telehealth sites and a better understanding of the potential interaction and connectivity across the various systems is required. Many Telehealth sites were installed for clinical use and cannot be readily adapted for other functions. Clinical services also remain the first priority for the use of these facilities. Informants noted that increasing demand for these resources from an expanding range of functions will require management.

These three Telehealth-funded projects successfully applied videoconferencing to support professional learning in population health in NSW. They were well received and together reached over 500 people in metropolitan and rural locations. Together they confirmed how increased access to training and professional development opportunities for a substantial number of rural and remote workers can be achieved with few resources. They demonstrated the potential of videoconferencing to deliver e-learning initiatives and how this technology can be used to build and sustain learning communities that provide support to remote practitioners.

The projects demonstrated the range of complexity in the delivery of learning that is possible; from information sessions to competency-based modules that included the assessment of participants.³ The participation of experts in the sessions generated interest in these projects; previous evaluations of the delivery of Bug Breakfast in NSW has shown that remote participants especially value the opportunity to interact with experts.^4,5 Videoconferencing should however be viewed as only one vehicle for providing capacity building and the mode of delivery chosen to deliver a learning session should fit both the purpose of a session and the resources available.⁶

The geographical distance spanned by these projects was large. For example, the largest rural AHS, the Greater Western, covers 55% of the landmass of NSW.⁷ While the majority of the NSW population live in urban settings, 20% live in inner regional areas and 7% in outer regional and remote areas.⁸ The delivery of population health capacity building activities to health staff across this large geographical area was made possible by using videoconferencing as the delivery method.

Videoconferencing a learning session is different to the delivery of face-to-face sessions and requires preparation and support.^9,10 Combining videoconferencing with other, related learning supports ensured a good quality episode of training and the evaluation confirmed some of the other elements that need to be in place for this to occur. These are: central project coordination and local site facilitators; reliable information technology systems in rural and remote areas, including ready access to computers and printers; a supportive environment encouraged by senior staff; and an appropriate level of infrastructure support at the state and local level.

A strategic review of Telehealth in NSW Health undertaken in 2008 suggested that Telehealth could provide a vehicle for developing an e-learning environment for the NSW Health system. Indeed, a recent survey conducted by the NSW Department of Health in 2009 identified 148 e-learning programs currently delivered through Telehealth.

The NSW Health Telehealth Network is a large network and consequently offers a means of providing equity of access to information and learning and development activities. In addition, Telehealth is not the only telecommunications network in NSW. A mapping exercise describing all the major networks that can be accessed by NSW Health staff for videoconferencing, and the potential connectivity between these systems, could expand the resources available for e-learning activities.

The evaluation has demonstrated that the invitation for submissions through the Telehealth Innovation Fund 2005–2006 provided an important stimulus for population health services to consider how they might incorporate Telehealth in their practice. Until 2005, there had been few funding submissions from the public health field. The inclusion of population health interventions in the 2005–2006 invitation has generated enthusiasm for applying the technologies available through Telehealth by the public health workforce in NSW.

The evaluation confirmed the value of videoconferencing as one vehicle through which equity of access to learning opportunities for population health workers across NSW can be achieved. It also demonstrated how it can be used to build and sustain learning communities to support remote practitioners.