Evaluation of virtual accreditation of medical specialist training sites for ophthalmology in Australia and New Zealand during the COVID-19 pandemic
Santosh Khanal![https://orcid.org/0000-0003-2688-1355](/media/client/orcid_16x16.png)
A Royal Australian and New Zealand College of Ophthalmologists, 94–98 Chalmers Street, Surry Hills, NSW 2010, Australia.
Australian Health Review 46(4) 453-459 https://doi.org/10.1071/AH22031
Submitted: 26 April 2021 Accepted: 1 May 2022 Published: 31 May 2022
© 2022 The Author(s) (or their employer(s)). Published by CSIRO Publishing on behalf of AHHA. This is an open access article distributed under the Creative Commons Attribution 4.0 International License (CC BY).
Abstract
Objective To evaluate the suitability and acceptability of virtual training post accreditation visits conducted online for medical specialist training in ophthalmology in Australia and New Zealand.
Methods A two-phase study (pilot and implementation) was conducted. In the pilot phase, an open-ended observation proforma was used by the authors to independently record their observations, which were later compared and discussed until consensus was achieved. All participants were asked to complete an online survey. A document analysis of accreditation documents was conducted. Observation data were broken down into themes and triangulated with online survey and document analysis results. In the implementation phase, the inspections were observed by one of the authors (SK) and the observation notes were discussed with other authors to obtain a contextual and consensual view. A document analysis of all accreditation-related documentation was undertaken. The documents included in the document analysis were planning and scheduling records, interview and inspection notes, training post inspection fact and document notices and accreditation reports. Finally, a post-inspection focus group of all inspectors was conducted.
Results The accreditation interviews adequately addressed all relevant issues with high levels of robustness and reliability. Participants found it more difficult to discuss complex issues virtually compared with onsite visits. The virtual accreditation reports were not any different to what would be expected if a face-to-face accreditation visit had been conducted; however, it was not possible using the virtual inspection to determine the appropriateness of facilities and clinic layout to support and facilitate trainee learning and supervision.
Conclusions Virtual accreditation of training posts in medical specialist training is viable in limited circumstances where there are no known complex training post-related issues and the site has not made substantial changes to clinic and theatre layout, equipment and facilities since the previous accreditation.
Keywords: accreditation, medical specialist training, ophthalmology, post supervisor, postgraduate medical education, trainee, training post, training site.
Introduction
Accreditation of healthcare and training systems is used as a quality assurance and control mechanism internationally to ensure that the health services and settings have the ability to train future practitioners at an appropriate standard to provide optimal patient care.1 In the context of postgraduate medical training in Australia and New Zealand, the bi-national specialist medical colleges are accredited by the Australian Medical Council to deliver postgraduate medical training in both countries.2 As a condition of this accreditation, the specialist medical colleges are required to set standards and accredit the training posts.
The Royal Australian and New Zealand College of Ophthalmologists (RANZCO) postgraduate ophthalmology training is a 5-year program that compares favourably with similar programs in other developed countries.3 The program is delivered in three stages (basic, advanced, and final year) in Australia and New Zealand by RANZCO’s eight training networks. There are training posts within a training network, such as hospitals and private practices, where the trainees are rotated to (Table 1). Training posts are classified as regional if they are based in locations categorised between 2 and 7 under the Modified Monash Model geographical classification in Australia4 and outside the metropolitan areas of Auckland, Wellington and Christchurch in New Zealand.
![]() |
RANZCO has developed accreditation standards for basic and advanced training posts.5 All training posts offering basic and advanced training need to be accredited before commencing and on a 3-yearly cycle, generally based on geography. The accreditation team consists of two ophthalmologists (one highly experienced senior inspector and another inspector) from different training networks to the one being inspected and one RANZCO staff member, and the process is governed by the Training Post Inspectorate.
Virtual accreditation
Use of virtual technology is well-established in medical education and training. Specific examples in ophthalmology include remote supervision of trainees,6,7 which is included in the RANZCO accreditation standards,5 surgical simulation training,8,9 which will be a mandatory component of the RANZCO training program from 2022, and online selection and examinations to minimise the risk of avoidable exposure to coronavirus disease 2019 (COVID-19) for examiners and trainees.
Other sectors such as engineering10 are increasingly accrediting educational facilities virtually, but there are very few examples in medical education. Recently, the Oman Medical Specialty Board was remotely accredited by the Accreditation Council for Graduate Medical Education International.11 In the US, guidelines have been released for US nursing colleges seeking virtual accreditation12 after the Department of Education allowed virtual site visits for accreditation in 2020 due to COVID-19 interruptions.13
At RANZCO, we felt that virtual accreditation could be an alternative approach to reduce backlog of overdue accreditations due to COVID-19 travel restrictions since March 2020; however, for virtual accreditation to be deemed a suitable alternative to face-to-face accreditation, it was essential that the integrity, credibility, and the high standards of face-to-face visits were maintained. We conducted a pilot phase followed by an implementation study to evaluate whether virtual accreditation could adequately identify and resolve training post-related issues.
Methods
We included a total of 75 training posts in the study: two in the pilot and 73 in implementation phases (Table 1). The pilot training posts (one in regional Australia and one in regional New Zealand) were selected opportunistically because their accreditations were due, and these training posts did not require visits to multiple locations. In the implementation phase, training posts were recruited as they came up for accreditation based on their geographical location during the study period (April–November 2021). It is noted that all authors have previously conducted face-to-face accreditation visits.
The virtual accreditation workflow and timelines are presented in Fig. 1. Data collection methods used in this study were observation; online qualitative survey and document analysis; and focus groups of inspectors.
![]() |
Observation
All three authors observed all the interviews and inspections with their video and audio turned off at the pilot phase. One of the authors (SK) observed the interviews and inspections in the implementation phase. An open-ended observation proforma was used to address the following key areas:
Alignment of interviews and inspection accreditation standards
Non-verbal aspects of the meetings
Incidental and opportunistic discussions around training
Engagement of the participants
Overall reliability of the processes
The investigators took notes independently; these were later compared and discussed until consensus was achieved.
Online survey
All participants in the pilot phase, including training post supervisors, trainees and members of the accreditation team were asked to complete a qualitative online survey to further understand their perception of virtual accreditation visits and to explore what did and did not work. Feedback was also sought on how the virtual accreditation visits could be made more effective. A qualitative online survey was considered more appropriate for the participants than semi-structured interviews because of the flexibility of an online survey and its previous successful use by RANZCO in seeking qualitative feedback from a similar cohort.
The online survey questions were developed based on the observation findings to further explore the concordance between the participants' and investigators’ insight on the value and usefulness of the virtual accreditation visits in the context of RANZCO accreditation standards and policy imperatives.
Document analysis
A document analysis of all the standard documentation that were collated and prepared by the inspections team as part of the virtual accreditation process was conducted and an assessment of their contribution to the accreditation decision- making process was undertaken. These were then compared with the documentation from previous face-to-face accreditation visits for the same training posts to assess whether similar levels of scrutiny and oversights were applied.
Focus group of RANZCO inspectors
Following the completion of both phases of the study and interim analysis of the results, a focus group discussion of all RANZCO inspectors was organised, which was facilitated by the Chief Inspector (GG). The main purpose of the focus group was to share the inspectors’ experiences and discuss the usefulness and acceptability of virtual inspections compared to face-to-face accreditation visits and determine the most suitable approach going forward.
Ethics approval
Ethics approval for this study was obtained from the RANZCO Human Research Ethics Committee.
Results
Observation – interviews
We observed that the interviews aligned with the accreditation standards and that all areas were appropriately addressed at the same standards as that of face-to-face interviews.
Inspectors probed into the issues as they were brought up and were able to obtain substantial and credible feedback, which they were then able to verify with other interviewees. The issues identified on the accreditation application forms and those raised by participants were discussed extensively. There did not appear to be any barriers to the inspection team asking appropriate questions to obtain insights about the training posts and probe into issues, and we also did not observe any reservations or hesitations from any of the interview participants in expressing their views and proposing remedial actions. The structured and formal nature of the virtual interviews, however, appeared to be somewhat prohibitive in facilitating collegiality between the inspectors and the local Fellows and trainees, as there were almost no opportunities for casual conversations.
Observation – inspection
The virtual inspection was not at all satisfactory where the site’s facilities and equipment required inspections.
During an inspection of a training post during the pilot phase, the internet feed dropped in and out when the post supervisor used their mobile devices to show videos of the outpatient clinics, theatre rooms and the hospital emergency department. The inspection team was unable to determine how conducive the setting would be for the trainees and the supervisors to interact between themselves when both were seeing patients in different rooms, or assess whether the ophthalmic equipment and facilities in the emergency departments were adequate.
Document analysis
We found the application and outcome documents and the accreditation reports of the virtual accreditation visits similar to the previous face-to-face ones, and there was no evidence to suggest that the content of the interim and final reports would have been any different if the visits were face-to-face as opposed to virtual. The level of detail in the virtual accreditation reports was on par with the previous face-to-face visit reports.
Online survey
The online survey response rate was 81.8% (9/11; two post supervisors, three trainees, three inspectors and one RANZCO staff), of which six (66.7%) had been involved in a face-to-face accreditation visit before.
None of the respondents who had previously been involved in face-to-face accreditation found the virtual accreditation to be better than face-to-face in any of the aspects in which feedback was sought (Fig. 2). None of the respondents were opposed to a virtual format for future accreditation if their concerns could be addressed, although it was pointed out by 60% of the respondents (3/5) that the virtual interviews were not conducive to building rapport between the inspectors and the interviewees.
![]() |
Information technology (IT)-related issues were faced by four respondents (36.4%). As highlighted by one inspector, the main concerns regarding IT were that the actual inspection of the department facilities could not be completed satisfactorily due to connectivity dead zones in the hospital; there were infrequent instances when the quality of video streaming dropped during interviews causing incoherent responses.
Implementation phase
All inspection teams were able to complete their allocated inspections virtually with none recommending a re-inspection, in person, of the sites.
Of the 73 training posts that were inspected, the inspection teams were able to identify various administrative issues in five posts (6.9%), such as overbooked clinics, limited administrative and clinical support for trainees and lack of continuity of supervision. Equipment and facilities at one training post (1.4%) were found to be inadequate. The virtual accreditations were also able to identify workplace-related issues at seven sites (9.9%) that warranted immediate further action, which included bullying and harassment (two posts, 2.8%) and non-adherence to RANZCO accreditation requirements for the number of supervised clinic and theatre sessions (five posts, 7.0%).
As appropriate, all the above issues were raised, discussed and appropriately actioned with the Head of Department, hospital executives, or Director/s of Training at the network so that they complied with RANZCO policies.
During the implementation phase, we did not encounter any IT-related issues during the interviews. However, given our poor experience with the video inspection of a site during the pilot phase, we did not attempt to undertake video inspection and asked the posts to send photographs of the equipment and facilities including the trainee consultation room. Although the photographs helped the accreditation teams determine what equipment and facilities were available, we found this approach deficient in helping understand how conducive the layout was to facilitate interaction between the trainees and their supervisor.
The feedback from participants about virtual accreditation was largely positive. All respondents felt that they were able to adequately provide input and suggestions and respond to issues raised by the inspection teams. Trainees indicated that they felt comfortable participating in the virtual interviews, although they did not seem to have a particular preference for one or the other, which was also the case for hospital executives and administrative staff. The few supervisors and other ophthalmologists who expressed a preference for face-to-face inspections did so to strengthen the collegiality between RANZCO and local Fellows.
Focus group of inspectors
In the focus group, the inspectors expressed mixed views about virtual accreditation. It was acknowledged that virtual accreditation offered some benefits, particularly around flexibility, cost-savings and environmental sustainability; however, inspectors experienced difficulties in building rapport and using visual cues. Importantly, it was noted by two inspectors that trainees were more forthcoming in the virtual interviews than the previous face-to-face ones they had conducted. Inspectors also suggested that they found it advantageous for the inspection team to have a rapid and confidential interim discussion in between interviews to develop a common understanding of issues and how best to address them in subsequent interviews.
It was suggested that when the inspections were conducted virtually, RANZCO lost one of the very few opportunities to develop collegiality with rural and remote ophthalmologists who are less likely to be involved in college activities. Further, the inspectors suggested that they found it more challenging to advocate with the local health administrators for enhanced ophthalmic services or resources when the meetings were conducted virtually as opposed to face-to-face.
In the context of travel, there were clearly mixed opinions, where some inspectors valued the opportunity to travel as a perk of their voluntary role whereas others did not express a specific preference. A suggestion was made that a hybrid model be used where one or two members of the accreditation team travel to the site with the others joining virtually, with one inspector pointing out that this would meet RANZCO’s needs for accreditation inspections while also contributing to reducing the environmental footprint of inspection-related travel.
Discussion
Based on our observations and document analysis, we believe contentious issues were discussed adequately and professionally in the virtual interviews and resolutions were achieved. However, the inspectors suggested it was more difficult than in face-to-face interviews to build rapport and rely on body language and other non-verbal cues. Similar findings have been reported in studies that piloted web-based selection for postgraduate medical training.14,15 Recent evidence shows that virtual meetings and interviews are increasingly being accepted by stakeholders16 and that appropriate planning, thoughtful implementation and deliberate engagement with inspectors and other participants from an early stage can largely mitigate the real and perceived limitations of virtual interviews.17
It does not appear that virtual inspection of the training facilities would be of much help in making accreditation decisions. We found that virtual inspections, particularly in large settings such as hospitals and eye centres with multiple subspecialities, do not provide much useful information to help develop an understanding of patient flow, access of equipment to trainees or trainee–supervisor interaction in the clinics and theatre. Health service facilities, including equipment, undoubtedly influence the quality of medical training1,18 and therefore it is important that the inspection of sites are not undermined in any way. Virtual inspection may be doable for smaller sites such as community-based clinics and outreach centres where trainees are rotated to from their main posting, but it is unlikely to work for larger eye centres and hospitals.
Although we did not quantify cost savings in our study, virtual accreditation could be reasonably expected to reduce the cost of accreditation for the accrediting body and training posts given the relatively high costs of face-to-face accreditation activities.19 It has been reported that virtual site visits of academic institutions20 and sonography diagnostic centres21 have yielded similar outcomes to face-to-face visits at much lower costs. Another important advantage of virtual accreditation is its contribution to limiting the environmental footprint of the healthcare sector22 by reducing avoidable travel.23 RANZCO has strong commitments towards environmentally sustainable practice, as demonstrated by endorsing and co-signing a report on the threat of climate change to health care,24 and supporting environmentally sustainable practices in the eye health sector.25
Decisions regarding face-to-face or virtual accreditation visits need to consider other intangible benefits. Importantly, invaluable opportunities for collegial interactions between the inspectors, representing the College, and their fellow specialists are reduced with virtual accreditation. Collegiality is highly valued by medical professionals to foster patient care and health advocacy, and has traditionally been facilitated by professional bodies.26 The inspection visits also allow RANZCO to advocate to hospital boards and regulatory authorities for improved ophthalmic services and resources in underserviced areas and disadvantaged communities. If accreditation is to be conducted virtually, other mechanisms will have to developed to fill this local advocacy gap. Another issue to consider is the voluntary nature of the role of training post inspectors and their personal motivations for active and continual engagement.27 Sitting in front of a computer all day for a few consecutive days could be a deterrent. If virtual accreditation is to be made workable, it would be important to seek a balance, perhaps by using a hybrid approach where inspectors are given the choice to complete a mix of both face-to-face and virtual accreditations.
A limitation of our study could be said to be that we did not assess patient perception and experience of the services at the training posts. This is a direct reflection of the existing processes for RANZCO training post accreditation, which does not include patient feedback, rather than a study limitation. Also, an observation we made during this study was that how similar the training post accreditation requirements were between specialist medical colleges in Australia and New Zealand, yet there seems to be very little, if any, collaboration between the colleges. Further, we were unable to find any information on the public domain about how these standards have been maintained and monitored by all colleges, including RANZCO. We believe this is a missed opportunity for the medical colleges and the organisations that host the training posts in developing efficiencies by sharing learnings and resources.
Our study has enabled us to better understand the intricacies of the training post accreditation and the value it serves to ophthalmology training in Australia and New Zealand. Although virtual accreditation is unlikely to fully replace face-to-face, there could be a role for virtual interviews post-COVID. Including virtual interviews in training post accreditation will facilitate input from a broader range of stakeholders, which will further strengthen accreditation outcomes. The ability to partake in both face-to-face and virtual accreditation will preserve the attractiveness of training post inspector roles for the ophthalmologists who volunteer.
Data availability
The data that support this study cannot be publicly shared due to ethical or privacy reasons and may be shared upon reasonable request to the corresponding author if appropriate.
Conflicts of interest
The authors declare no conflicts of interest.
Declaration of Funding
This research did not receive any specific funding.
References
[1] World Health Organization. Quality and accreditation in health care services. A global review Evidence and Information for Policy Department of Health Service Provision. Geneva: World Health Organization; 2003.[2] Medical Board of Australia. Specialist medical colleges. 2010. Available at https://www.medicalboard.gov.au/Accreditation/Specialist-medical-colleges.aspx
[3] Chan WH, Saedon H, Falcon MG. Postgraduate ophthalmic training: how do we compare? Eye 2011; 25 965–967.
| Postgraduate ophthalmic training: how do we compare?Crossref | GoogleScholarGoogle Scholar | 21833031PubMed |
[4] Australian Government Department of Health. Modified Monash Model. Australian Government Department of Health; 2020. Available at https://www.health.gov.au/health-topics/rural-health-workforce/classifications/mmm
[5] Royal Australian and New Zealand College of Ophthalmologists. Standards for Ophthalmology Training Posts Includes: Code of Best Practice. Royal Australian and New Zealand College of Ophthalmologists; 2018. Available at https://ranzco.edu/wp-content/uploads/2019/07/Standards_for_ophthalmology_training_posts.pdf
[6] Wearne S, Dornan T, Teunissen PW, Skinner T. Twelve tips on how to set up postgraduate training via remote clinical supervision. Med Teach 2013; 35 891–894.
| Twelve tips on how to set up postgraduate training via remote clinical supervision.Crossref | GoogleScholarGoogle Scholar | 23782042PubMed |
[7] Cameron M, Ray R, Sabesan S. Remote supervision of medical training via videoconference in northern Australia: a qualitative study of the perspectives of supervisors and trainees. BMJ Open 2015; 5, e006444.
| Remote supervision of medical training via videoconference in northern Australia: a qualitative study of the perspectives of supervisors and trainees.Crossref | GoogleScholarGoogle Scholar |
[8] Gillan SN, Saleh GM. Ophthalmic Surgical Simulation: A New Era. JAMA Ophthalmol 2013; 131 1623–1624.
| Ophthalmic Surgical Simulation: A New Era.Crossref | GoogleScholarGoogle Scholar | 24337557PubMed |
[9] Lee R, Raison N, Lau WY, et al. A systematic review of simulation-based training tools for technical and non-technical skills in ophthalmology. Eye 2020; 34 1737–1759.
| A systematic review of simulation-based training tools for technical and non-technical skills in ophthalmology.Crossref | GoogleScholarGoogle Scholar | 32203241PubMed |
[10] Hussain W, Spady WG, Tayyab Naqash M, et al. ABET Accreditation during and after COVID19 - Navigating the Digital Age. IEEE Access 2020; 8 218997–219046.
| ABET Accreditation during and after COVID19 - Navigating the Digital Age.Crossref | GoogleScholarGoogle Scholar | 34976567PubMed |
[11] Al-Bualy R, Al Mjeni RAM, Al Abri H, Al Sinani S. Graduate Medical Education Accreditation Across Borders: Toward a Remote Alternative. J Grad Med Educ 2021; 13 26
| Graduate Medical Education Accreditation Across Borders: Toward a Remote Alternative.Crossref | GoogleScholarGoogle Scholar | 33680296PubMed |
[12] Cobourne K, Shellenbarger T. Virtual site visits: a new approach to nursing accreditation. Teach Learn Nurs 2021; 16 162–165.
| Virtual site visits: a new approach to nursing accreditation.Crossref | GoogleScholarGoogle Scholar |
[13] American Psychological Association COVID-19 update and site visits. APA; 2020. Available at https://www.apa.org/ed/accreditation/newsletter/2020/09/site-visits
[14] Daram SR, Wu R, Tang SJ. Interview from anywhere: Feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process. Am J Gastroenterol 2014; 109 155–159.
| Interview from anywhere: Feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process.Crossref | GoogleScholarGoogle Scholar | 24496418PubMed |
[15] Shah SK, Arora S, Skipper B, et al. Randomized evaluation of a web based interview process for urology resident selection. J Urol 2012; 187 1380–1384.
| Randomized evaluation of a web based interview process for urology resident selection.Crossref | GoogleScholarGoogle Scholar | 22341282PubMed |
[16] Robinson KA, Shin B, Gangadharan SP. A Comparison Between In-Person and Virtual Fellowship Interviews During the COVID-19 Pandemic. J Surg Educ 2021; 78 1175–1181.
| A Comparison Between In-Person and Virtual Fellowship Interviews During the COVID-19 Pandemic.Crossref | GoogleScholarGoogle Scholar | 33250429PubMed |
[17] Wolff M, Burrows H. Planning for Virtual Interviews: Residency Recruitment During a Pandemic. Acad Pediatr 2021; 21 24
| Planning for Virtual Interviews: Residency Recruitment During a Pandemic.Crossref | GoogleScholarGoogle Scholar | 33068812PubMed |
[18] Medical Board of Australia. Postgraduate medical councils. 2022. Available at https://www.medicalboard.gov.au/accreditation/postgraduate-medical-councils.aspx
[19] Mumford V, Greenfield G, Hodgen A, et al. Counting the costs of accreditation in acute care: an activity-based costing approach. BMJ Open 2015; 5 e008850
| Counting the costs of accreditation in acute care: an activity-based costing approach.Crossref | GoogleScholarGoogle Scholar | 26351190PubMed |
[20] McEwan A, Walsh Jr E. Virtual Site Visits: The Future of Institutional Accreditation? In: Barton S, Hedberg J, Suzuki K, editors. Proceedings of Global Learn 2011. Association for the Advancement of Computing in Education (AACE); 2011. pp. 605–612.
[21] ProQuest. The Impact of Virtual Site Visits on Programmatic Accreditation through the Commission on Accreditation of Allied Health Education Programs (CAAHEP). 2021. Available at https://www.proquest.com/openview/ef42b237258191e26996bd2f08b69345/1?pq-origsite=gscholar&cbl=18750&diss=y
[22] Lenzen M, Malik A, Li M, et al. The environmental footprint of health care: a global assessment. Lancet Planet Heal 2020; 4 e271–e279.
| The environmental footprint of health care: a global assessment.Crossref | GoogleScholarGoogle Scholar |
[23] Storz MA. A Practical Guide for Physicians and Health Care Workers to Reduce Their Carbon Footprint in Daily Clinical Work. Perm J 2018; 22 17–145.
| A Practical Guide for Physicians and Health Care Workers to Reduce Their Carbon Footprint in Daily Clinical Work.Crossref | GoogleScholarGoogle Scholar | 29616907PubMed |
[24] RACP. Climate change is the biggest threat to health system. RACP; 2020. Available at https://www.racp.edu.au/news-and-events/media-releases/climate-change-is-the-biggest-threat-to-health-system/
[25] Royal Australian and New Zealand College of Ophthalmologists. Climate Emergency declared by IAPB on Earth Day 2021. RANZCO; 2021. Available at https://ranzco.edu/news/climate-emergency-declared-by-iapb-on-earth-day-2021/
[26] Laws Jr ER. The advantages of collegiality. Bull N Y Acad Med 1992; 68 297–302.
[27] Bussell H, Forbes D. Understanding the volunteer market: the what, where, who and why of volunteering. Int J Nonprofit Volunt Sect Mark 2002; 7 244–257.
| Understanding the volunteer market: the what, where, who and why of volunteering.Crossref | GoogleScholarGoogle Scholar |