Evaluation of virtual accreditation of medical specialist training sites for ophthalmology in Australia and New Zealand during the COVID-19 pandemic

Keywords: accreditation, medical specialist training, ophthalmology, post supervisor, postgraduate medical education, trainee, training post, training site.

Accreditation of healthcare and training systems is used as a quality assurance and control mechanism internationally to ensure that the health services and settings have the ability to train future practitioners at an appropriate standard to provide optimal patient care.¹ In the context of postgraduate medical training in Australia and New Zealand, the bi-national specialist medical colleges are accredited by the Australian Medical Council to deliver postgraduate medical training in both countries.² As a condition of this accreditation, the specialist medical colleges are required to set standards and accredit the training posts.

The Royal Australian and New Zealand College of Ophthalmologists (RANZCO) postgraduate ophthalmology training is a 5-year program that compares favourably with similar programs in other developed countries.³ The program is delivered in three stages (basic, advanced, and final year) in Australia and New Zealand by RANZCO’s eight training networks. There are training posts within a training network, such as hospitals and private practices, where the trainees are rotated to (Table 1). Training posts are classified as regional if they are based in locations categorised between 2 and 7 under the Modified Monash Model geographical classification in Australia⁴ and outside the metropolitan areas of Auckland, Wellington and Christchurch in New Zealand.

Table 1.  Number of RANZCO-accredited training posts by network and regionality in 2022 and the number included in the study (both pilot and implementation phases).

RANZCO has developed accreditation standards for basic and advanced training posts.⁵ All training posts offering basic and advanced training need to be accredited before commencing and on a 3-yearly cycle, generally based on geography. The accreditation team consists of two ophthalmologists (one highly experienced senior inspector and another inspector) from different training networks to the one being inspected and one RANZCO staff member, and the process is governed by the Training Post Inspectorate.

Use of virtual technology is well-established in medical education and training. Specific examples in ophthalmology include remote supervision of trainees,^6,7 which is included in the RANZCO accreditation standards,⁵ surgical simulation training,^8,9 which will be a mandatory component of the RANZCO training program from 2022, and online selection and examinations to minimise the risk of avoidable exposure to coronavirus disease 2019 (COVID-19) for examiners and trainees.

Other sectors such as engineering¹⁰ are increasingly accrediting educational facilities virtually, but there are very few examples in medical education. Recently, the Oman Medical Specialty Board was remotely accredited by the Accreditation Council for Graduate Medical Education International.¹¹ In the US, guidelines have been released for US nursing colleges seeking virtual accreditation¹² after the Department of Education allowed virtual site visits for accreditation in 2020 due to COVID-19 interruptions.¹³

At RANZCO, we felt that virtual accreditation could be an alternative approach to reduce backlog of overdue accreditations due to COVID-19 travel restrictions since March 2020; however, for virtual accreditation to be deemed a suitable alternative to face-to-face accreditation, it was essential that the integrity, credibility, and the high standards of face-to-face visits were maintained. We conducted a pilot phase followed by an implementation study to evaluate whether virtual accreditation could adequately identify and resolve training post-related issues.

We included a total of 75 training posts in the study: two in the pilot and 73 in implementation phases (Table 1). The pilot training posts (one in regional Australia and one in regional New Zealand) were selected opportunistically because their accreditations were due, and these training posts did not require visits to multiple locations. In the implementation phase, training posts were recruited as they came up for accreditation based on their geographical location during the study period (April–November 2021). It is noted that all authors have previously conducted face-to-face accreditation visits.

The virtual accreditation workflow and timelines are presented in Fig. 1. Data collection methods used in this study were observation; online qualitative survey and document analysis; and focus groups of inspectors.

Fig. 1.  Virtual accreditation workflow and timeline. Activities in bold were conducted differently to face-to-face accreditation visits.

All three authors observed all the interviews and inspections with their video and audio turned off at the pilot phase. One of the authors (SK) observed the interviews and inspections in the implementation phase. An open-ended observation proforma was used to address the following key areas:

1. Alignment of interviews and inspection accreditation standards

2. Non-verbal aspects of the meetings

3. Incidental and opportunistic discussions around training

4. Engagement of the participants

5. Overall reliability of the processes

The investigators took notes independently; these were later compared and discussed until consensus was achieved.

All participants in the pilot phase, including training post supervisors, trainees and members of the accreditation team were asked to complete a qualitative online survey to further understand their perception of virtual accreditation visits and to explore what did and did not work. Feedback was also sought on how the virtual accreditation visits could be made more effective. A qualitative online survey was considered more appropriate for the participants than semi-structured interviews because of the flexibility of an online survey and its previous successful use by RANZCO in seeking qualitative feedback from a similar cohort.

The online survey questions were developed based on the observation findings to further explore the concordance between the participants' and investigators’ insight on the value and usefulness of the virtual accreditation visits in the context of RANZCO accreditation standards and policy imperatives.

A document analysis of all the standard documentation that were collated and prepared by the inspections team as part of the virtual accreditation process was conducted and an assessment of their contribution to the accreditation decision- making process was undertaken. These were then compared with the documentation from previous face-to-face accreditation visits for the same training posts to assess whether similar levels of scrutiny and oversights were applied.

Following the completion of both phases of the study and interim analysis of the results, a focus group discussion of all RANZCO inspectors was organised, which was facilitated by the Chief Inspector (GG). The main purpose of the focus group was to share the inspectors’ experiences and discuss the usefulness and acceptability of virtual inspections compared to face-to-face accreditation visits and determine the most suitable approach going forward.

Ethics approval for this study was obtained from the RANZCO Human Research Ethics Committee.

We observed that the interviews aligned with the accreditation standards and that all areas were appropriately addressed at the same standards as that of face-to-face interviews.

Inspectors probed into the issues as they were brought up and were able to obtain substantial and credible feedback, which they were then able to verify with other interviewees. The issues identified on the accreditation application forms and those raised by participants were discussed extensively. There did not appear to be any barriers to the inspection team asking appropriate questions to obtain insights about the training posts and probe into issues, and we also did not observe any reservations or hesitations from any of the interview participants in expressing their views and proposing remedial actions. The structured and formal nature of the virtual interviews, however, appeared to be somewhat prohibitive in facilitating collegiality between the inspectors and the local Fellows and trainees, as there were almost no opportunities for casual conversations.

The virtual inspection was not at all satisfactory where the site’s facilities and equipment required inspections.

During an inspection of a training post during the pilot phase, the internet feed dropped in and out when the post supervisor used their mobile devices to show videos of the outpatient clinics, theatre rooms and the hospital emergency department. The inspection team was unable to determine how conducive the setting would be for the trainees and the supervisors to interact between themselves when both were seeing patients in different rooms, or assess whether the ophthalmic equipment and facilities in the emergency departments were adequate.

We found the application and outcome documents and the accreditation reports of the virtual accreditation visits similar to the previous face-to-face ones, and there was no evidence to suggest that the content of the interim and final reports would have been any different if the visits were face-to-face as opposed to virtual. The level of detail in the virtual accreditation reports was on par with the previous face-to-face visit reports.

The online survey response rate was 81.8% (9/11; two post supervisors, three trainees, three inspectors and one RANZCO staff), of which six (66.7%) had been involved in a face-to-face accreditation visit before.

None of the respondents who had previously been involved in face-to-face accreditation found the virtual accreditation to be better than face-to-face in any of the aspects in which feedback was sought (Fig. 2). None of the respondents were opposed to a virtual format for future accreditation if their concerns could be addressed, although it was pointed out by 60% of the respondents (3/5) that the virtual interviews were not conducive to building rapport between the inspectors and the interviewees.

Fig. 2.  Percentage of respondents who had been involved in previous accreditations (n = 6) who found the virtual accreditation to provide ‘just the same’ opportunities as face-to-face. The remaining respondents selected ‘worse’ as their response. None of the respondents selected the other three available options (better, much better, much worse) for any of the items.

Information technology (IT)-related issues were faced by four respondents (36.4%). As highlighted by one inspector, the main concerns regarding IT were that the actual inspection of the department facilities could not be completed satisfactorily due to connectivity dead zones in the hospital; there were infrequent instances when the quality of video streaming dropped during interviews causing incoherent responses.

All inspection teams were able to complete their allocated inspections virtually with none recommending a re-inspection, in person, of the sites.

Of the 73 training posts that were inspected, the inspection teams were able to identify various administrative issues in five posts (6.9%), such as overbooked clinics, limited administrative and clinical support for trainees and lack of continuity of supervision. Equipment and facilities at one training post (1.4%) were found to be inadequate. The virtual accreditations were also able to identify workplace-related issues at seven sites (9.9%) that warranted immediate further action, which included bullying and harassment (two posts, 2.8%) and non-adherence to RANZCO accreditation requirements for the number of supervised clinic and theatre sessions (five posts, 7.0%).

As appropriate, all the above issues were raised, discussed and appropriately actioned with the Head of Department, hospital executives, or Director/s of Training at the network so that they complied with RANZCO policies.

During the implementation phase, we did not encounter any IT-related issues during the interviews. However, given our poor experience with the video inspection of a site during the pilot phase, we did not attempt to undertake video inspection and asked the posts to send photographs of the equipment and facilities including the trainee consultation room. Although the photographs helped the accreditation teams determine what equipment and facilities were available, we found this approach deficient in helping understand how conducive the layout was to facilitate interaction between the trainees and their supervisor.

The feedback from participants about virtual accreditation was largely positive. All respondents felt that they were able to adequately provide input and suggestions and respond to issues raised by the inspection teams. Trainees indicated that they felt comfortable participating in the virtual interviews, although they did not seem to have a particular preference for one or the other, which was also the case for hospital executives and administrative staff. The few supervisors and other ophthalmologists who expressed a preference for face-to-face inspections did so to strengthen the collegiality between RANZCO and local Fellows.

In the focus group, the inspectors expressed mixed views about virtual accreditation. It was acknowledged that virtual accreditation offered some benefits, particularly around flexibility, cost-savings and environmental sustainability; however, inspectors experienced difficulties in building rapport and using visual cues. Importantly, it was noted by two inspectors that trainees were more forthcoming in the virtual interviews than the previous face-to-face ones they had conducted. Inspectors also suggested that they found it advantageous for the inspection team to have a rapid and confidential interim discussion in between interviews to develop a common understanding of issues and how best to address them in subsequent interviews.

It was suggested that when the inspections were conducted virtually, RANZCO lost one of the very few opportunities to develop collegiality with rural and remote ophthalmologists who are less likely to be involved in college activities. Further, the inspectors suggested that they found it more challenging to advocate with the local health administrators for enhanced ophthalmic services or resources when the meetings were conducted virtually as opposed to face-to-face.

In the context of travel, there were clearly mixed opinions, where some inspectors valued the opportunity to travel as a perk of their voluntary role whereas others did not express a specific preference. A suggestion was made that a hybrid model be used where one or two members of the accreditation team travel to the site with the others joining virtually, with one inspector pointing out that this would meet RANZCO’s needs for accreditation inspections while also contributing to reducing the environmental footprint of inspection-related travel.

Based on our observations and document analysis, we believe contentious issues were discussed adequately and professionally in the virtual interviews and resolutions were achieved. However, the inspectors suggested it was more difficult than in face-to-face interviews to build rapport and rely on body language and other non-verbal cues. Similar findings have been reported in studies that piloted web-based selection for postgraduate medical training.^14,15 Recent evidence shows that virtual meetings and interviews are increasingly being accepted by stakeholders¹⁶ and that appropriate planning, thoughtful implementation and deliberate engagement with inspectors and other participants from an early stage can largely mitigate the real and perceived limitations of virtual interviews.¹⁷

It does not appear that virtual inspection of the training facilities would be of much help in making accreditation decisions. We found that virtual inspections, particularly in large settings such as hospitals and eye centres with multiple subspecialities, do not provide much useful information to help develop an understanding of patient flow, access of equipment to trainees or trainee–supervisor interaction in the clinics and theatre. Health service facilities, including equipment, undoubtedly influence the quality of medical training^1,18 and therefore it is important that the inspection of sites are not undermined in any way. Virtual inspection may be doable for smaller sites such as community-based clinics and outreach centres where trainees are rotated to from their main posting, but it is unlikely to work for larger eye centres and hospitals.

Although we did not quantify cost savings in our study, virtual accreditation could be reasonably expected to reduce the cost of accreditation for the accrediting body and training posts given the relatively high costs of face-to-face accreditation activities.¹⁹ It has been reported that virtual site visits of academic institutions²⁰ and sonography diagnostic centres²¹ have yielded similar outcomes to face-to-face visits at much lower costs. Another important advantage of virtual accreditation is its contribution to limiting the environmental footprint of the healthcare sector²² by reducing avoidable travel.²³ RANZCO has strong commitments towards environmentally sustainable practice, as demonstrated by endorsing and co-signing a report on the threat of climate change to health care,²⁴ and supporting environmentally sustainable practices in the eye health sector.²⁵

Decisions regarding face-to-face or virtual accreditation visits need to consider other intangible benefits. Importantly, invaluable opportunities for collegial interactions between the inspectors, representing the College, and their fellow specialists are reduced with virtual accreditation. Collegiality is highly valued by medical professionals to foster patient care and health advocacy, and has traditionally been facilitated by professional bodies.²⁶ The inspection visits also allow RANZCO to advocate to hospital boards and regulatory authorities for improved ophthalmic services and resources in underserviced areas and disadvantaged communities. If accreditation is to be conducted virtually, other mechanisms will have to developed to fill this local advocacy gap. Another issue to consider is the voluntary nature of the role of training post inspectors and their personal motivations for active and continual engagement.²⁷ Sitting in front of a computer all day for a few consecutive days could be a deterrent. If virtual accreditation is to be made workable, it would be important to seek a balance, perhaps by using a hybrid approach where inspectors are given the choice to complete a mix of both face-to-face and virtual accreditations.

A limitation of our study could be said to be that we did not assess patient perception and experience of the services at the training posts. This is a direct reflection of the existing processes for RANZCO training post accreditation, which does not include patient feedback, rather than a study limitation. Also, an observation we made during this study was that how similar the training post accreditation requirements were between specialist medical colleges in Australia and New Zealand, yet there seems to be very little, if any, collaboration between the colleges. Further, we were unable to find any information on the public domain about how these standards have been maintained and monitored by all colleges, including RANZCO. We believe this is a missed opportunity for the medical colleges and the organisations that host the training posts in developing efficiencies by sharing learnings and resources.

Our study has enabled us to better understand the intricacies of the training post accreditation and the value it serves to ophthalmology training in Australia and New Zealand. Although virtual accreditation is unlikely to fully replace face-to-face, there could be a role for virtual interviews post-COVID. Including virtual interviews in training post accreditation will facilitate input from a broader range of stakeholders, which will further strengthen accreditation outcomes. The ability to partake in both face-to-face and virtual accreditation will preserve the attractiveness of training post inspector roles for the ophthalmologists who volunteer.

The data that support this study cannot be publicly shared due to ethical or privacy reasons and may be shared upon reasonable request to the corresponding author if appropriate.

The authors declare no conflicts of interest.

This research did not receive any specific funding.