The re-evaluation of the use of videoconferencing to deliver the Bug Breakfast, December 2004

In July 2002, the videoconferencing of Bug Breakfast was first evaluated to document the quality of the delivery and identify ways to improve the learning environment. Participants described recurrent problems, which included the size of the room at the live site and, for the remote sites, the quality of the sound. A series of recommendations to improve delivery were published as part of the report describing the evaluation. These were systematically implemented.

In December 2004, the videoconferencing of the Bug Breakfast was re-evaluated to assess whether the implementation of these recommendations had changed the quality of the learning environments.

Eleven sites were requested to participate in the session and all were successfully connected. Eight sites elected to receive the session via videoconference and three requested an audio transmission only, which was also delivered through the videoconferencing system. All sites participated in the evaluation. The audience comprised: 41 people at the live site and 57 people between the remote sites. The overall response rate to the questionnaire was 91%.

Live site participants at North Sydney primarily occupied public health roles, whereas the remote audience had roughly equal proportions of clinical and public health professionals. The combined remote audience was larger than the live audience.

The North Sydney audience were satisfied with the layout of the rooms for the delivery of the session and offered few comments regarding the potential for further improvement. The majority of the remote site audience reported that the picture and sound quality was either good or average. These findings demonstrate that the actions taken to improve the delivery at the live site and the transmission of sound to the remote sites had improved the quality of the learning environment for both audiences. All participants valued the Bug Breakfast: as a forum for learning and sharing of up-to-date information; for providing access to a range of communicable disease topics; and for the opportunity to network with colleagues across NSW.

The Bug Breakfast provides a model for applying videoconferencing to the communication and professional development needs of the public health workface. Remote practitioners value videoconferencing because it allows them the opportunity to participate in this continuing professional development activity.

Since this evaluation further actions have been taken to refine this model of delivery.

Bug Breakfast is a series of hour-long breakfast seminars that have been delivered since 1990 by the Population Health Division at the NSW Department of Health. Held once a month, each session examines the public health aspects of a communicable disease of current relevance. Since 1999, these sessions have been made accessible to a wider audience located at rural sites across NSW by videoconferencing through the resources of the NSW Telehealth Initiative.

In July 2002, an evaluation of the videoconferencing of the Bug Breakfast was undertaken to assess the quality of the delivery at both the live and remote sites and to identify ways to enhance the learning environment for all participants. The findings highlighted consistent problems experienced by the remote audience, in particular with the quality of the sound, while the venue used to deliver the live presentation at North Sydney was no longer large enough to accommodate the growing audience.

A series of recommendations to improve the delivery were subsequently outlined in the report ‘An evaluation of videoconferencing Bug Breakfast’.¹ Through the collaboration of the Department’s Telehealth Unit and the Public Health Training and Development Branch many recommendations were promptly actioned. For example, the delivery of Bug Breakfast was moved to a larger suite of conference rooms at the NSW Department of Health in North Sydney to allow the session to be presented in a theatrette style. This arrangement provides the live site audience at North Sydney with significantly improved accommodation. It also reduces the ambient noise in the delivery of the session because presenters now use a lectern with a directional microphone.

A working party comprising representatives from the Communicable Diseases Branch, the Public Health Training and Development Branch and the NSW Public Health Unit Directors Forum was convened to determine actions to address the remaining recommendations.

This evaluation was undertaken to reassess the quality of the learning experience for the audience at the live and remote sites. It sought to determine whether the implementation of the recommendations from the 2002 evaluation had resulted in improvements to the learning environments for both audiences. Table 1 summarises the action that had been taken to implement each recommendation at the time of the re-evaluation.

Table 1.  Summary of the outcomes of recommendations arising from the 2002 evaluation of Bug Breakfast at the time of the re-evaluation

The December 2004 session of Bug Breakfast was evaluated. The session was entitled ‘The Rash: A current investigation of an outbreak among methadone users’.

The evaluation was implemented at short notice as an amalgamation of the area health services in NSW was planned for January 2005. The immediate effect of these changes on the delivery of Bug Breakfast was unknown and we sought to ensure that the environment for the re-evaluation closely resembled that of the evaluation in 2002.

The procedure for delivering Bug Breakfast was:

• the live site was the suite of conference rooms (Kurraba, Taronga and Tumbalong) at the NSW Department of Health, North Sydney

• the session was advertised by email to staff within the Population Health Division at the NSW Department of Health, to the public health units within the area health services of NSW and to a small number of public health professionals working in other organisations e.g. Work Cover, University of Sydney and Department of Defence

• Trainee Public Health Officers from the NSW Public Health Officer Training Program assisted with the organisation and delivery of the session

• the videoconference was coordinated by the NSW Telehealth Initiative

• Telstra Conferencing was the external provider who dialled the sites and provided the bridge

• the connection speed was 256 kbps (medium bandwidth)

• the videoconference was ‘voice activated’

• the session was one hour long

• there were three presenters with a question and answer session at the conclusion of the presentations

• electronic copies of the PowerPoint presentations were supplied by the presenters and distributed in advance by email to the remote sites

• at each remote site a person within the public health unit acted as a facilitator and was responsible for: promoting the Bug Breakfast session locally; and receiving and distributing copies of the PowerPoint presentations.

The usual procedure is for the PowerPoint presentations to be loaded onto a laptop that is connected to the videoconferencing system. This allows the presentations to be digitally transmitted directly to the remote sites. The PowerPoint slides appear on the screen of the remote sites videoconferencing unit accompanied by the presenters’ voice.

On the day of the evaluation the PowerPoint presentations were unable to be loaded onto the laptop. So, to enable the remotes sites to view the slides, the camera at the live site videoconferencing unit was focussed on the screen where the presentations were being projected for the live site audience. Consequently, remotes sites were able to view either the presenter or the slides on their screen, which was managed by the North Sydney site.

Eleven remote sites requested a connection to the session. Of these, eight sites elected to receive the session via videoconference and three sites (Bathurst, Liverpool and Wallsend) requested an audio transmission only. The audio transmission was also delivered through the videoconferencing system. All eleven sites were successfully connected on the morning.

The evaluation in 2002 included four groups: the participants; the Bug Breakfast facilitators at each remote site; those involved in organising the session; and the presenters. As the purpose of the 2004 re-evaluation was to determine any perceived changes to the quality of the learning environments at the live and remote sites, only the participants were surveyed on this occasion.

The two questionnaires (one for the live site participants and one for the remote site participants) developed for the 2002 evaluation were slightly modified for the 2004 evaluation. Questions that were included in the 2002 questionnaires that explored issues other than the learning environment were removed. For example, information about travel times to venues was not collected. Otherwise, the questions remained the same to allow comparison.

A copy of the questionnaires used in the 2004 evaluation is included in Appendices 1 and 2. The questionnaires were administered in the same way as in 2002.¹ At the beginning of the session, the chair advised the audience that the session would be evaluated at the end of the transmission and sought the cooperation of the audience. Participation was voluntary. The questionnaires were distributed and collected at the end of the session at all sites.

Facilitators at each remote site assisted by:

• counting the number of participants at their site

• distributing the questionnaires

• collecting the completed questionnaires

• sending the questionnaires to the investigators at North Sydney.

Responses were entered into a Microsoft Access database and then imported into SAS (version 8.02) for analysis. A quantitative analysis was carried out for the close-ended responses. The responses to the open ended questions and the interviews were analysed for major themes.

There were 47 people present at the live site in North Sydney. Of these, six – the three investigators and three presenters – were not required to complete a questionnaire. There were a total of 57 participants at the 11 remote sites. In addition, two people who are regular participants of Bug Breakfast but who were unable to attend this session – one from Albury and one from Port Macquarie – provided feedback based on their previous experience. While they have not been included in the response rate for the remote sites, issues raised in their responses are included in the discussion.

A total of 89 participants completed a questionnaire, a response rate of 91%. The number of people attending the session and the number of participant responses from each location are presented in Table 2. The response rates from the remote sites and the North Sydney site were 96% and 83%, respectively.

Table 2.  Number of participants and the number of responses for all sites that participated in the Bug Breakfast evaluation, 2004

The age and gender of the respondents is presented in Table 3. The North Sydney audience had almost equal representation of men and women. The remote audience contained more women and was older compared to the North Sydney audience.

Table 3.  Age and gender of participants by site in the Bug Breakfast evaluation, 2004

The North Sydney audience were primarily engaged in public health roles, whereas the remote sites had similar proportions of clinical and public health professionals (see Table 4). Fifty-six per cent (n = 19) of the North Sydney audience were involved with training in some role, either with delivery or as a trainee. This compared with 7% for the remote audience. The North Sydney audience included trainees from programs including the NSW Public Health Officer Training Program, the NSW Biostatistical Officer Training Program and the Advanced Training Scheme offered by the Australasian Faculty of Public Health Medicine.

Table 4.  Type of job and core role within job of participants by site in the Bug Breakfast evaluation, 2004

Eighty-two per cent of the North Sydney audience had previously attended Bug Breakfast compared with 62% of remote participants.

Thirty-one per cent (n = 17) of remote site participants indicated that the videoconferencing technology hindered their learning experience compared to only 6% (n = 2) of the participants from North Sydney. Comments were provided by 22 respondents from the remote audience and these were primarily related to dissatisfaction with the sound (n = 6) and the picture quality (n = 10).

The layout of the suite of rooms at the North Sydney site for the transmission of Bug Breakfast was reported to be satisfactory by 97% (n = 33) of the audience at North Sydney. Three per cent (n = 1) of respondents, however, reported that the environment hindered their learning experience.

Seventy-six per cent of respondents indicated that sufficient time was allowed for questions. Comments were elicited from 10 participants; of these, six requested that additional time should be allocated for questions.

Of the 34 remote site participants who had previously attended a Bug Breakfast, 25 (74%) indicated that the videoconferencing quality of the session was typical.

Twenty-one participants (62%) reported technical problems ‘sometimes’ occurred, whereas seven participants reported that they occurred ‘frequently’.

The majority of remote site participants reported that the sound quality was either ‘good’ or ‘average’ (Table 5).

Table 5.  Sound quality reported by remote site participants attending Bug Breakfast videoconference, 2004

Of the eight remote sites that received the picture and audio broadcast through the videoconferencing system, the majority of remote site participants reported the picture quality as either ‘good’ or ‘average’ (Table 6).

Table 6.  Picture quality reported by remote site participants attending Bug Breakfast videoconference, 2004

Seventy-six per cent (n = 42) of participants reported having access to a copy of the PowerPoint presentation prior to the session.

Seventeen per cent of participants (n = 15) recalled taking part in the previous evaluation in 2002. Participants were asked to recall any changes that they had observed in the delivery of Bug Breakfast since the initial evaluation. Comments were elicited from 15 participants, with the North Sydney audience citing improvements in the venue (n = 4) and the remote audience citing improvements with the delivery through the technology (n = 3).

Participants were asked about what they liked most about attending Bug Breakfast. This elicited 33 and 52 responses from the live and remote sites respectively. The things they liked were: learning and sharing of information (n = 30); up-to-date information (n = 16); range of communicable disease topics (n = 10); and networking (n = 7).

Participants were asked about what they least liked about attending Bug Breakfast, the nature of the responses from the North Sydney and remote sites differed. Five participants from North Sydney disliked the early start at 8.30am, whereas the remote audience cited technical problems (n = 12) and poor sound quality (n = 6).

The response rate to this evaluation compares favourably with the response rate for the 2002 evaluation (93%) and allows broad comparisons to be made between the findings.

The 2004 evaluation showed that participants at North Sydney were satisfied with the new venue and other changes made to the delivery (Table 7). In 2002, half the audience felt that the environment hindered learning and in 2004 only one person felt this.

Table 7.  Summary of findings regarding changes made to the delivery of Bug Breakfast from the 2002 and 2004 evaluations for the live site participants

On both occasions, there was dissatisfaction with the amount of time available for questions (Table 7). Comments in 2004 included ‘would like more time for questions’ and ‘remote sites get too much preference’.

The videoconference is allocated one hour and does not extend beyond this. Fifteen minutes is allowed at the end of the session for questions and, consequently, if the presentations encroach upon this, question time is reduced. Further questions from the live site are invited after questions are taken from the remote sites. This evaluation confirmed that the participants value the opportunity to ask questions and that question time should be protected within the session.

Table 8 presents the remote audience ratings for sound quality from both evaluations. In 2002, the quality of the sound for the broadcast was reported as ‘poor’ by most remote site participants.¹ In 2004, 16% of remote site participants reported ‘poor’ sound quality, suggesting that the wide range of interventions had improved the quality of the sound for the majority. Poor sound quality was randomly reported across the remote sites, a finding that was difficult to interpret.

Since the 2002 evaluation, before each session begins, the speakers are routinely reminded to speak into the microphone located on the lectern to improve the transmission of the audio for remote site participants. A laptop is placed on the lectern so that speakers can read their slides while looking towards the camera and speaking into the microphone. However, audio fading can occur when presenters turn away from the lectern to look at their presentation on the screen that is behind them or if they move away from the lectern while speaking.

Table 8.  Summary of findings for picture and sound quality from the 2002 and 2004 evaluations of Bug Breakfast for remote sites

The picture quality was reported as either ‘good’ or ‘average’ by 95% of the remote audience, similar to the results from the 2002 evaluation (see Table 8).¹ This finding was surprising, as we had expected that the picture quality of the PowerPoint presentation would be suboptimal on the day of the evaluation as the usual procedure for delivering the PowerPoint presentation could not be followed. The set up followed usually results in a poorer picture quality of the presentation for remote sites due to the movement of the screen and the light from the data projector.

The picture quality of the PowerPoint presentations is also influenced by the style and format used by the presenter. It is difficult for remote audiences to distinguish text on PowerPoint presentations that use complex backgrounds, particularly on this occasion with the set-up of the camera on the projected screen.

The session topic, the investigation of an outbreak of a rash in methadone users that was underway at that time, was of particular interest to the drug and alcohol workforce. Among the audience at the remote sites, one-third identified their main workforce responsibility as ‘drug and alcohol’ compared with 10% of the North Sydney audience. Members of the drug and alcohol workforce are not regular attendees of Bug Breakfast and this finding suggests that the Bug Breakfast audience is not static and attendance is influenced by the session topic. For example, the 2002 Bug Breakfast topic was Meningococcal Disease and this was of interest to health professionals responsible for immunisation and communicable diseases.

There is, however, a core group of participants who attend regularly. In this evaluation, 72% of participants reported that they had previously attended a Bug Breakfast session; this was similar to the findings in the 2002 evaluation. Despite the high number of participants previously reporting attending a Bug Breakfast session, only 17% of participants could recall taking part in the 2002 evaluation. However, participants that reported participating in the earlier evaluation were able to identify the changes that had been implemented to improve the delivery.

The Centre for Health Protection at the NSW Department of Health provides the financial assistance that allows Bug Breakfast to be videoconferenced.

Connections to the rural area health services have been limited to one site in each area and this number is restricted by cost. However, the expanding interest in Bug Breakfast has resulted in numerous requests for additional sites to connect. For the 2002 evaluation, 10 sites requested a connection, in 2004 the maximum number of sites that were dialled out to was 11.

To meet the request by additional sites to join, sites have recently been offered the option of dialling into the session. However, the number of dial in and dial out sites taking part in Bug Breakfast has been closely monitored, as the maximum number of sites that can be linked by a single videoconference bridge is 20. While it is possible to link multiple bridges, this can affect the stability and quality of the connection for all participants.

Overall, participants supported Bug Breakfast as a professional development activity for the delivery of communicable diseases information. Participants enjoyed receiving ‘up to date current information’ and found ‘videoconferencing a useful tool to communicate with distance audiences’. Remote site participants valued the opportunity to participate and referred to the program as an essential part of professional development for public health professionals residing in rural areas of NSW. For the 2002 evaluation, participants provided detailed comments on the delivery, whereas there were few comments in this evaluation. This may reflect a general satisfaction with the improved quality of delivery of the Bug Breakfast session.

This evaluation of the videoconferencing of Bug Breakfast confirmed the findings from the 2002 evaluation that the public health workforce values Bug Breakfast as a continuing professional development activity. This finding is reflected in the continuing demand from new sites to join the session. The implementation of the recommendations following the 2002 evaluation has resulted in improvements in the quality of the learning experience for both the live site and remote participants, with both audiences reporting high satisfaction with sound and picture quality. Issues remain regarding protecting time for questions and determining the number of sites that it is feasible to connect to a Bug Breakfast seminar.

The NSW Telehealth Initiative has enabled Bug Breakfast to be regularly and continuously delivered via videoconferencing to public health professionals working in rural and remote New South Wales for nine years. The findings of this evaluation demonstrate that it is possible to progressively improve the quality of the delivery of training via videoconferencing through cycles of quality review.

Since the evaluation, several actions have been carried out to improve the delivery of Bug Breakfast.

The Public Health Training and Development Branch has:

• briefed speakers on delivering presentations using videoconferencing

• developed a Bug Breakfast website on the NSW Health Intranet where facilitators register attendance of their remote site and are able to access copies of the speakers’ presentations and provide feedback on the session

• established a generic email account for Bug Breakfast to provide a central point of contact for administration purposes

• included the Telehealth Helpdesk in the link up on occasions to observe and monitor technical configurations for quality assurance purposes

• accommodated additional sites as dial in sites

• published a Bug Breakfast Delivery Manual that describes how to deliver Bug Breakfast.²

We would like to thank the following people for their assistance in the evaluation of the December 2004 Bug Breakfast:

Jill Hanvey, Kimberlee Isaac and Deborah Oong.

Ala’a Al Murieb, Megan Black, Kym Bush, Kerry Chant, Greg Cheguelman, Margaret Lesjak, Peter Massey, Marita O’Callaghan, Maggi Osbourn, Gail Osborne and Janet Terry.

Centre for Health Protection for financial support for the delivery of Bug Breakfast. Jeremy McAnulty for overseeing the content.

Many thanks to all participants who completed the 2004 evaluation questionnaire.