Pilot study of using a web and teleconference for the delivery of an Epi Info training session to public health units in NSW, 2005

The evaluation was carried out in two parts: participants were asked to complete a questionnaire; and a focus group was held with the three site facilitators following the session.

There were 12 participants and the response rate to the questionnaire was 100%.

After the training session, 83% of participants felt able to apply Epi Info to a foodborne disease outbreak investigation. The participants reported an increase in their confidence in developing a questionnaire using Epi Info, entering data and conducting basic analyses of that data. However, there was little change in their confidence in interpreting the results of these analyses.

Participants were asked about the delivery of the session. The majority of the participants reported that the picture quality of the PowerPoint presentation on their computer screens was either ‘good’ or ‘average’. However, the sound quality of the teleconference was reported to be of ‘average’ or ‘poor’ quality.

During the Epi Info session, two sites experienced disconnections from the web conference and several participants reported delays in the PowerPoint slides being displayed on their computer screen. These technical issues affected the delivery of the training for all the participants. Despite these difficulties, participants cited benefits in receiving training this way, principally being able to access training without the need to travel.

Facilitators commented on the support and organisation provided by the NSW Department of Health on setting up the session. They could identify opportunities in using web conferencing; for example, for continuing professional development activities and working with public health colleagues. However, disconnections during the training session affected the quality of the learning experience.

This pilot demonstrated that professional development activities can be delivered via a combined web conference and teleconference, and identified simple measures that can improve the learning environment. However, the technical issues raised should be explored and resolved before promoting this means of delivery widely.

In October 2004, the Communicable Diseases Branch of the NSW Department of Health conducted a Foodborne Diseases Workshop. As part of this workshop, an introductory session on Epi Info version 3.3 was offered to public health professionals. The session was well attended and well received. After the workshop, the Food Portfolio Group of the NSW Public Health Directors Forum approached the Communicable Diseases Branch and described a need for public health professionals in NSW to be trained in the use of Epi Info to assist with the management of outbreak investigations. Epi Info is a public domain software package developed by the Centers for Disease Control and Prevention to be used in outbreak investigations for the management and analysis of data.¹ Epi Info allows public health professionals to create questionnaires rapidly, enter data and carry out analysis to generate statistics and graphs to help identify the likely cause of an outbreak.¹

The Communicable Diseases Branch in association with the Executive Officer for the Public Health Directors Forum subsequently sought to organise an introductory Epi Info training session for a number of public health units in New South Wales. At the time of this study, March 2005, there were 17 Public Health Units in NSW; one located in each of the former area health services. Consequently, the people to receive the training were geographically dispersed. The preferred delivery format was an interactive session where the features of Epi Info could be demonstrated using a practical example of a foodborne disease outbreak.

Consequently a training session was developed to enable the remote delivery of training using a combination of a web conference supported by a teleconference. This allowed public health professionals to receive the training at their worksite. This pilot of the training allowed for both the efficacy of the training and the medium through which it was delivered to be evaluated.

Web conferencing is computer based and delivered over the Internet between two or more people situated at different locations. The rationale for using web conferencing was that it enabled a speaker’s PowerPoint presentation to be delivered to an audience in real time using Internet Protocol (IP) network connections.² Web conferencing has numerous features, these include: sharing a presentation; document or white board; text chat; tools for annotating documents; and attendance indictors.^2,3 The features used for the delivery of the Epi Info training were restricted to sharing a presentation and the attendance indictor.^A The reason for restricting the functions was to ensure that the focus of the session was the Epi Info training and that the web conference did not detract from the learning experience.

Combining the web conference with a teleconference allowed active conversation and voice instruction between the trainers and the sites. It is possible to web conference without teleconference support, the interaction between participants and trainers then occurs through the ‘chat’ facility and is a typed interaction. The teleconference was considered necessary for the pilot because at each site there were a number of participants who shared the computer connected to the web conference, restricting their access to the chat facility through which they could ask questions.

To evaluate the pilot of the delivery of an Epi Info training session to three public health units in NSW via a web conference supported by a teleconference.

1. To introduce public health workers to Epi Info and its application to the investigation of outbreaks of foodborne illness.

2. To evaluate the learning outcomes associated with the training session.

3. To increase the confidence and skills of presenters in using web conferencing for training purposes.

1. To evaluate the participants response to using web and teleconferencing for training.

2. To evaluate the quality of the delivery of a training session through a combined web and teleconference.

3. To determine the technical and structural resources required to participate in the web conference.

1. To determine potential broader applications of web and teleconferencing for public health practice.

For this pilot the Mid Western, Western Sydney and Greater Murray Public Health Units received the training from the NSW Department of Health where a group of three organisers and trainers were based. Members of the Food Portfolio Group volunteered their sites for the pilot. As most people are unfamiliar with receiving training through web conferencing, a training session in how to web conference was also provided and this preceded the Epi Info training session.

The method is described in three parts:

1. preparing the sites to receive the Epi Info training session

2. delivering the training

3. evaluating the training session.

The following steps were followed:

A facilitator was identified at each of the three public health units. The facilitator’s role was to prepare their site for the web conference so that they could access the Epi Info session. The facilitators were provided with instructions on how to install the software Meeting Manager on the computer/s to be used for the web conference. All sites successfully installed this program.⁴

The objective of this training was to provide participants with a basic understanding of the functions of web conferencing and for them to gain confidence in using it as a medium for the delivery of training. Telstra Conferencing using WebEx was the web conferencing provider used for both the web conferencing training and the Epi Info training session. A representative from Telstra facilitated the web conferencing training and also arranged a teleconference to support the web conference. The training was scheduled for one hour and the content covered is described in Box 1.

Facilitators were responsible for the setup and organisation of their site for the Epi Info session. Their responsibilities included ensuring that there was a suitable venue with the following resources to support delivery:

• a computer with Meeting Manager installed and connected to the web conference⁴

• a speakerphone connected to the teleconference

• a data projector to view the web conferencing computer screen

• a laptop or personal computer with Epi Info Version 3.3 installed for each participant.

The facilitators were provided with a checklist of these requirements.

The facilitators were also responsible for distributing the materials that were provided for both the training session and its evaluation, these included:

• an outline of the content to be covered

• a workbook for each participant (this included a copy of the PowerPoint presentation used)

• the evaluation questionnaire for participants

• a list of questions for the facilitators.

The Epi Info training session was conducted in early March 2005. All three public health units successfully joined to the teleconference and the web conference. Mid Western and Western Sydney Public Health Units each had one computer connected to the web conference, while at the Greater Murray Public Health Unit there were two connected (see Figure 1). The duration of the training session was two hours.

Prior to commencing the training, the organisers carried out a roll call and then requested that the public health unit sites mute their phones unless they were asking a question.

The trainer at the North Sydney site displayed a PowerPoint presentation on their computer screen and this was viewed by the participants at each of the public health unit sites on their computer linked to the web conference. During the training session, participants were encouraged to use their laptops or personal computers to complete the Epi Info exercises as instructed by the trainer.

The content of the training session was adapted from the Epi Info training manual developed by the Centers for Disease Control and Prevention. The training was based on a foodborne outbreak investigation and covered the following:

• creating questionnaires that automatically generate a database

• entering data into the database

• analysing the data to produce statistics and graphs.

The training was modified for delivery through web conferencing.

Both the effectiveness of the Epi Info training session in increasing the participants’ knowledge of and confidence in using Epi Info and their experience of using web conferencing for delivery were assessed.

Participants were asked to complete a questionnaire and the facilitators were invited to participate in a focus group that followed the session.

A structured questionnaire was developed that contained three types of questions:

• close-ended questions, including rating scales

• close-ended questions with the opportunity for comment

• open-ended questions.

The questionnaire had two parts. The first part was completed before the session started and sought to establish the participant’s previous experience of foodborne outbreak investigations and a self-assessed baseline of their confidence in using Epi Info for developing a questionnaire, entering data and conducting basic analyses. They were also asked about their confidence in interpreting the findings of an Epi Info analysis. Participants could indicate their degree of confidence by selecting from a horizontal scale consisting of five boxes; the first, marked 1, represented ‘not confident at all’ to the last, marked 5, ‘extremely confident’. The second part of the questionnaire was administered at the conclusion of the session. Their confidence in using Epi Info was reassessed using the same scale. Their experience of the web conference was sought, including whether they attended the training and the quality of the sound and the picture.

The facilitators distributed the questionnaire to participants at each site. The participant questionnaire is included in Appendix 1.

While the scales measuring confidence in using Epi Info were constructed to encourage ticks within the boxes, people also marked the line between the boxes. This was interpreted in all cases as halfway between two integers on the scale. The responses were entered into and analysed using a database created in Epi Info version 3.3. Descriptive statistics were used to analyse the responses to the close-ended questions. Open-ended responses were explored for major themes.

The three facilitators were invited to participate in a focus group with the organiser/trainers via teleconference following the Epi Info training session. Permission was sought from them to record the teleconference onto a compact disc. The teleconference was organised through Telstra Conferencing and Telstra recorded the compact disc. The purpose was to explore any issues associated with the training session or encountered in the organisation of the web conference.

The facilitators were provided with the questions to be used approximately one week before the Epi Info training session. Questions related to: their overall impression of the training; difficulties encountered in organising the session; the value in rolling out Epi Info training across the state via web conferencing; potential applications of web conferencing in their work; and their views on delivering continuing professional development activities to staff via web conferencing. Their responses were examined and the major themes that emerged identified. The list of questions is included in Appendix 2.

There were 12 participants across the three public health units. The number of people attending the training from each location is presented in Table 1. The response rate to the questionnaire was 100%.

Table 1.  Number of participants at each site for the Epi Info training session

The age and gender of participants is presented in Table 2. The majority of participants (83%) were female. Participants were asked their job title and four participants reported that they were communicable diseases surveillance officers, two participants were immunisation coordinators and the remaining participants were responsible for management, research and office administrative functions. Almost all participants reported that they had previously had some involvement in a foodborne outbreak investigation. The responsibilities that participants assumed at the time of an outbreak investigation included: conducting interviews; data entry and analysis; site investigation and providing advice on infection control; and the collection of samples.

Table 2.  Age and gender of participants in the Epi Info training session n = 12

Eleven participants indicated that they had previously used Epi Info. Of these, three considered that their level of knowledge in using it was satisfactory whereas eight reported that it was poor.

Participants were asked to rate their confidence in using Epi Info to develop a questionnaire for a foodborne disease outbreak before and after the training. Figure 2 presents the before and after scores for the group and shows an increase in the participants’ self-reported confidence in creating a questionnaire.

Participants were asked to rank their confidence in entering data into a questionnaire created in Epi Info before and after training. Figure 3 shows their confidence increased, with two participants reporting to be extremely confident in performing this task.

Figure 4 illustrates the participants’ confidence before and after training in conducting basic analysis using Epi Info. Participants’ reported that their confidence increased. However, when participants were asked to rank how confident they felt in interpreting the analysis, Figure 5 shows that there was little difference in the scores before and after training.

Overall, 83% of participants felt confident enough to apply Epi Info to foodborne disease outbreak investigations in the future, with 17% of participants stating that they felt unsure.

None of the participants had previously used web conferencing. Six (50%) participants attended the web conferencing training. Of these six, five reported that it prepared them for the delivery of the Epi Info training by providing an overview of the technology.

On the day of the Epi Info training session participants were asked to rate the quality of the picture on the computer screen for the web conference. The majority of the participants reported it as either good or of average quality as presented in Table 3.

Table 3:  Picture quality of web conference reported by 12 participants across three sites

Participants were asked if the PowerPoint slide visible on their computer screen was the same slide that the presenter at the North Sydney site was describing. Four participants reported delays in the slide becoming visible on their screen, with one participant unsure if they were experiencing a delay. Responses to this question were inconsistent, as not all participants at the same site reported experiencing the delays, despite viewing the same computer screen in two of the public health units. This makes it difficult to interpret these results.

Participants were asked to rate the quality of the sound from the teleconference. Overall the sound was considered to be of average or poor quality, see Table 4. Participants’ perception regarding the quality of sound varied at each of the sites.

Table 4:  Sound quality of teleconference reported by 12 participants across three sites

Sixty-seven per cent (n = 8) of participants reported that, other than the quality of sound and picture, there were aspects of the web conferencing that detracted from the training. The Mid Western Public Health Unit reported that the web conference disconnected twice during the session whereas in the Greater Murray, one of the two computers used for the web conference also lost its connection. These disconnections slowed the pace of the training and this affected the sites that did not experience these technical difficulties.

Participants were asked what they liked most about participating in Epi Info training via web conferencing. Eleven participants responded to this question, with four citing the benefits of having training onsite and not needing to travel. Other comments included being able to carry out practical exercises with colleagues during the session.

Participants were also asked what they least liked about participating in the training and eight participants responded to this question. The majority of responses were focussed on technical issues such as sound quality and web conference disconnections. Despite the technical issues, only one participant indicated that they would not participate in a web conference for a continuing professional development activity, whereas a further four were unsure.

Several people took the opportunity to make additional comments about the pilot and the application of web conferencing and teleconferencing to public health. One person suggested that public health professionals could use these to obtain assistance in applying Epi Info in the event of an outbreak investigation. However, several people highlighted issues that would need to be addressed in order to provide training via a combined web and teleconference. These included: access to the Internet in regional areas; local site setup for training purposes; and access to teleconferencing facilities.

Participants were asked if they had any suggestions on how the training could be improved. Participants suggested allocating additional time for the delivery of the training on analysis and interpretation of results in Epi Info (n = 2), whereas other participants commented on the technical issues of web conferencing (n = 2) and the sound quality of the teleconference (n = 1).

This evaluation of using web and teleconferencing to deliver training was performed on a small pilot study involving 12 public health professionals on one occasion of training. The content of the session and the organisation of its delivery were carefully undertaken. Consequently these results provide an insight into some of the issues associated with providing professional development activities through these communication techniques as an alternative to face-to-face delivery for a selected and motivated group of participants. The major limitation of this pilot was, therefore, that there was only one training event and consequently the external variables that influence the quality of delivery could not be investigated in subsequent training sessions.

The participants reported that their feelings of confidence in applying Epi Info increased in three of the four parameters measured: developing a questionnaire; entering data; and conducting an analysis. This suggests that the Epi Info training session provided participants with an overview of the various functions of the software and therefore fulfilled one of the objectives of the training.

However, this did not extend to their confidence in interpreting findings. Possible explanations for this include: the way in which the training session was constructed and the time allowed for discussion of the results of the analysis. This finding may also reflect the participants pre-existing level of knowledge and skills in undertaking the interpretation of the results of quantitative data analyses. Those who had previous training in epidemiology may have had greater confidence in the analysis component.

The level of skill attained by participants was not assessed. While participants reported improved confidence and familiarity in using Epi Info, this does not necessarily reflect appropriate use of the software.

Prior to this training, none of the participants had used web conferencing. This highlights the need to provide a session to introduce web conferencing. The willingness of half the participants to attend the web conferencing training suggests a high level of interest and motivation in this group and differs from the experiences reported in the literature. The literature reports low levels of participation when preparatory training is offered and that most people using web conferencing only come to ‘grips’ with technology at the time of the interaction.¹

Delivering training via web conferencing to a ‘virtual audience’ introduces its own unique set of challenges to the trainer. In particular, the absence of non-verbal cues makes it difficult to vary the pace of the presentation for the audience. To overcome this problem, the trainer arranged for one of the organisers of the session to work through the Epi Info exercises with the participants to gauge the time taken to complete the exercises. The literature confirms that the virtual group audience differs from its face-to-face counterparts and that there is a ‘new learning culture’ created by adopting computer technology.⁵

Additional time is also required for discussing the analysis and its implications in managing an outbreak. In view of the wide range of skill levels of public health professionals in using Epi Info, there was support for a staged approach to delivering training. A facilitator also reported that there would be value in having someone familiar with Epi Info available at each of the sites to assist participants during the training session. There was also a suggestion of strengthening the Epi Info training by embedding the training in an outbreak scenario with a focus on study design, analysis and interpretation of results.

Web conferencing enabled the delivery of the PowerPoint presentation from the North Sydney site to all participants in real time, the teleconference provided the audio component. Facilitators indicated that other functions of web conferencing such as ‘chat’ and ‘raise hand’ may provide less intimidating ways for participants to ask questions and also result in fewer interruptions for the whole group. This request may also be due to several participants experiencing poor sound quality from the teleconference. It is likely that these participants used a speakerphone rather than an audio conferencing system. Other factors that will influence the quality of the sound are the set up of the training room and the proximity of participants to an audio conferencing system.

Two of the four computers from the three sites became disconnected from the web conference. This was unexpected, as disconnections were not experienced during the web conferencing training hosted by Telstra Conferencing. Participants also reported intermittent delays with the PowerPoint slide becoming visible. Subsequent inquiries with Telstra Conferencing and the Information Technology Branch at the NSW Department of Health advised that the bandwidth can influence the quality of the delivery for web conferencing. Having either a large numbers of users or a small number using functions requiring greater bandwidth can affect the delivery.⁶

This pilot therefore demonstrated that there are bandwidth implications for web conferencing over the current network.⁶ If there are low-speed links or network congestion, participants will receive a poorer delivery and this may affect the learning experience. Consequently, information technology departments within area health services need to be consulted regarding the bandwidth capacity of the network for web conferencing, particularly in remote areas.

This pilot has also confirmed that a suitable venue and appropriate equipment are required for the successful delivery of training. Where there are many participants at one site, a data projector is preferable to a computer monitor to display the web conference.

The number of sites connected to the web conference and the number of participants at each site also affects the format of the session. With increasing number of sites or participants, the potential for interaction with the presenter is reduced. Box 2 summarises the advice that would be provided to facilitators at sites to improve the quality of the learning environment experienced by participants.

Despite the technical difficulties encountered, over half the participants reported that they would participate in a web conference for a continuing professional development activity. Participants also suggested several applications of web conferencing to assist them in their role, these included: delivery of training; and carrying out their duties more efficiently and effectively.

This pilot demonstrated that an interactive training session could be delivered using a combined web and teleconference. Simple measures were identified that can improve the quality of the learning environment, including the layout of the room (such as the proximity to phones and computers). However, there are information technology issues that should be explored before promoting its wider use. Despite this, web conferencing has the potential to increase equity of access to training for both metropolitan and rural public health professionals.

1. Further pilots could be undertaken to support the development of web conferencing to deliver training, in particular to describe:

2. 1. types of continuing professional development activities that are suitable

   2. the implications on the network system and ways of managing this.

3. A series of guidelines could be produced that describe:

4. 1. how to organise continuing professional development activity via web conferencing

   2. how to use teleconferencing for a range of activities including advice on the technical requirements.

We would like to thank the NSW Telehealth Initiative Strategy for its financial support in the delivery of this pilot to the public health units. Many thanks to the members of the 2005 Food Portfolio Group of the NSW Public Health Directors Forum for their advice in the planning of this pilot and undertaking the role of facilitator at their respective public health units and finally to the participants, who took the time to complete the evaluation questionnaire.