Time to establish comprehensive long-term monitoring of Australian medical graduates?

Australia’s 20 medical schools have quite different approaches to selection and educational approaches. Some are postgraduate, others accept undergraduate entry and the duration of their courses varies: 4, 5 or 6 years. The effect of variation between schools on graduate ability and doctors’ long-term performance is unexplored, despite much strongly held opinion. Medical education has always been expensive for the community, and students themselves are now facing significant higher-education debts. The costs of the differences in course duration alone would suggest the need for some investigation, but there are more urgent issues.

There was recent acknowledgement by the 2015 Australian Review of Medical Intern Training¹ that medical graduates ‘…enter the health system highly qualified from a variety of…medical programs but with often limited experience in actual patient care and no baseline of work-ready capabilities.’

Other data also suggest all is not right longer term with the graduates of our medical schools: some are depressed and burnt out,^2,3 some are a source of patient complaints⁴ and many fail to practise evidence-based care.^5,6 This is despite Australian Health Practitioner regulatory requirements, both those associated with registration (self- and peer disclosure) and the professional practice standards described within Good Medical Practice.⁷

Some have advocated for a national licensing examination to ensure a more consistent standard of graduate. There is no such examination in Australia (unlike those that currently exist in the US and soon to exist in the UK), and instead there is a collaborative approach to assessing competency with common examination questions and examination stations.⁸ However, any examination system is limited because ‘…focus on measurable behaviours [may] ignore the kinds of higher-order thinking and acting that constitute competence in demanding work, such as medicine.’⁹ Skills rather than knowledge are modern priorities. Clinicians no longer need to remember vast numbers of facts and crucial skills now include translation of evidence, coordination of care and collaboration. The right attitudes and attributes are also required to excel in the use of such skills (e.g. self-reflection, motivation). Compared with knowledge, examination of skills is difficult, resource intensive and much more useful for students when formative (hence the call to move to ‘assessment for learning, not of learning’¹⁰).

Since 1985, medical schools have been supported and guided by the Australian Medical Council (AMC).¹¹ The AMC medical school accreditation process is one of self- and peer assessment against a set of broad standards. The AMC has encouraged diversity of approaches to medical education and its expert assessors provide an improvement-focused program assessment. US commentators recently criticised the effectiveness of this style of medical program accreditation as a stand-alone assurance of quality.¹² Yet, the AMC Standards for Assessment and Accreditation of Primary Medical Programs specifically require Australian schools to also track their graduates:

6.2.1 The medical education provider analyses the performance of cohorts of students and graduates in relation to the outcomes of the medical program.¹³

Providers currently have few tools available to assist them to do this. We propose that an approach to monitoring graduates long term be considered. The aim of such monitoring would be to improve the education students receive, tailoring it accurately based on Australian experiences and thus improve patient care. The proposed system has three elements: (1) ‘How do they feel?’; (2) ‘How are they doing?’; and (3) ‘How are their patients going?’ Explanation of these three elements and possibilities for monitoring follows.

The issue of poor doctor mental health, and its effects, has been publicised in Australian media recently with the tragic report of the suicide of three junior doctors.¹⁴ Issues cited include workplace culture, competition for training places, the financial cost of postgraduate examinations and the inability of the profession to look objectively at balancing career and life outside work.¹⁴ A recent Lancet editorial cited the fragmentation of clinical teams, shift working and loss of locus of control as contributors to stress and to the erosion of mechanisms that previously maintained resilience.¹⁵

The demonstrable extent of the problem of reduced well being and mental illness in students and doctors^2,3 supports long-term monitoring. Other countries do this. The Longitudinal Study of Norwegian Medical Students and Doctors¹⁶ has a major focus on well being and involves all Norwegian medical schools. It commenced in 1993 and is now reporting on 15-year follow-up work¹⁷. The UK has a National Training Survey (http://www.gmc-uk.org/education/surveys.asp accessed 3 August 2017); this compulsory survey is completed by those in post-primary medical qualification training and captures data on trainee satisfaction with their programs, compliance with General Medical Council standards and Royal College speciality curricula, data from speciality trainee annual progression reports and data from employers. The data are freely available online, allowing comparison of outcomes according to several domains, including qualifying medical school, geographical region, medical and surgical speciality, etc.

Australia has a Medical Schools Outcomes Database (MSOD)¹⁸, a national data collection established in 2004 by Medical Deans Australia and New Zealand (MDANZ) and recruiting over 90% of medical students (with over 30 000 participants now in the dataset). There are unique identifiers for participants and a linkage is currently being sought across the MSOD, medical registration and national health workforce datasets. The focus of the future linkage work is on career choice (e.g. intentions vs what happens). The questionnaire contains several interesting demographic and attitudinal items (including documented student concern about educational debt and satisfaction with medical schooling). The first cohort is now in their 5th year after graduation. The scope of the survey could be enhanced with the addition of items on career satisfaction, well being and mental health to create a more detailed and curriculum-relevant picture than is currently available. This work could even develop into a longitudinal study with the addition of items from the MSOD survey into the existing surveys that form part of Australian Health Practitioner Regulation Agency (AHPRA) registration processes. This could also provide an opportunity to look at issues around equity with regard to gender and career choices, time out of work, return to work etc. This would require agreements to share data and an understanding from students, doctors, health regulators and workforce planners around how the data will be used.

Actions are possible. The culture of performance and achievement that extends from high school to university and throughout a medical career is relentless, with the added stigma of perceived weakness if individuals ask for help or admit they are not coping.¹⁹ In theory, educational interventions that increase engagement,²⁰ improve students’ efficacy and resilience (see, for instance, Wald et al.²¹) and help them develop a positive sense of self²² can be protective against burn-out and depression. Griffith University investigated the association between training stress, coping skills and thoughts of dropping out of medicine in students and junior doctors.²³ Those serious about dropping out used avoidance and risky behaviour as coping mechanisms. In the UK, a tool based on established occupational health risk models has been developed to look at medical student well being.²⁴ When results of six medical schools were compared, schools did well in development of skills and knowledge, but scored more variably in areas such as work–life balance, academic demands and health. This formative tool can be used by medical schools to review their curricula and learning environment for quality improvement, and could be used to compare medical schools.

Discussions we have had with medical students and young doctors has included the importance of building support networks and self-monitoring of mental health and stress. Work clarifying the precise value of the many strategies that could be promoted, such as mindfulness, peer support groups, meditation, exercise and easy access to psychological services, is also needed. See Box 1 for more details.

Currently, Australian medical schools do not systematically and regularly engage in ‘post-market surveillance’ with regard to the competence and professional behaviours of their students. If they did, this would enable schools to improve their management of conduct and health issues, their approach to progression and potentially reconsider entry criteria.

There is much discussion on monitoring and influencing professional behaviours within medical school, which is beyond the scope of this paper (certainly the disorganised or those that resent direction often stay that way, hence the heralded association of failure to bring vaccination certificates on entry with later professional behaviour problems²⁵). To date, there has been little Australian research^26,27 that considers medical graduate performance in the workplace. However, it has been a recent focus in the UK, with studies of the ‘Foundation Year’.^28–30

In an ideal world, reliable junior doctor assessments would be able to be linked with medical school entry and assessment data. Unfortunately, New South Wales prevocational training assessments have been found to be superficial³¹ and Western Australian workplace assessments, of limited reliability.³² AHPRA is apparently concerned about the lack of the current utility of assessments for detecting problem doctors,³² and improved instruments and supervisor training seem likely. However, limited long-term investment in trainees by health services and supervisors (due to short-term rotations and contracts) will limit the effectiveness of any instrument.³³

Having a valid workplace assessment instrument alone will not be sufficient; it should be combined with study of successful graduates and enquiry into any perceived skill and capability deficits.^34,35 Medical schools may have significant support structures in place, in addition to policies and procedures that allow students with health issues to complete the course with reasonable adjustments. Students may expect these same supports in the workplace, and the realisation that this is often not the case is a shock. Medical schools need to work with students and health service employers to narrow the expectation–reality gap.

Managers of junior medical officers are keen for a feed-forward approach, with identification of issues that may mean junior doctors need more support. Cooperation and communication between employers, medical schools and possibly regulators could enable both feed-forward and feedback on doctors and trainees at risk (e.g. those with prior complaints or a lack of response to feedback). There are privacy issues in both directions, but these are not insurmountable when the overriding concerns are patient and doctor safety, and meetings allowing for frank two-way discussion could be organised. The creation of positive case studies could also help improve workplace support for doctors in health and well being. Finally, this linkage will ensure appropriate employer feedback into curriculum redesign.³⁵

There could also be behaviour and performance follow-up past the junior doctor years. Medical board sanctions are not a good measure, because these are rare and typically occur after years of harm or underperformance. Formal complaints (to healthcare complaints commissioners or AHPRA) are far more frequent and are often associated with poor communication by practitioners (skills that are teachable and assessable). Practitioner characteristics (e.g. age, sex, type of practice) form part of a recently developed predictive score for complaints,⁴ and the medical school attended could form part of such future analyses. See Box 2 for more details.

Patient outcomes are a key long-term measure. Many apparently competent graduates go on to provide suboptimal care to patients during their careers, especially by failing to practise evidence-based medicine. In one US study, up to 35% of end-of-life Medicare expenditures, and 12% of overall Medicare expenditures, were explained by physician beliefs, not justified either by patient preferences or by evidence of clinical effectiveness.³⁶ The Australian Atlases of Healthcare Variation (see https://www.safetyandquality.gov.au/atlas/atlas-2015/ and https://www.safetyandquality.gov.au/atlas/atlas-2017/, both accessed 3 August 2017) have demonstrated substantial variation in clinical practice, much of it unexplained. Doctors generate unnecessary tests and procedures⁵ and financially exploit patients.⁶ It is thought that ‘physician decision-making skills and perceptions of what constitutes evidence-based practice are influenced by the training they received in medical school’.³⁷ Regardless of what teaching may be provided with regard to current concepts of evidence-based medicine, curriculum practices that encourage self-directed learning have the potential to increase the ability of students to continue to learn well when they are employed, when they do need to identify their own educational deficits and fill them.³⁸

The effect of medical schooling on patient outcomes is too complex an intervention to study easily; many factors that determine the care the patient receives are not under the control of an individual provider, but affected by availability, patient preference, funding models, actions of other members of the health care team, etc. (the ‘dilution effect’).³⁹ This is particularly so for junior medical officers; thus, for this group, process measures such as teamwork competency offer more promise.

However, we do now have the resources and skills to undertake meaningful big data outcomes research on independent practitioners. Variation in patient outcomes related to education has been demonstrated for US speciality training programs. A study reflecting nearly 5 million deliveries, more than 4000 obstetricians and more than 100 residency programs found programs were associated with substantial variation in maternal complication rates, such that women treated by obstetricians who trained in programs with the highest complication rate had a complication rate one-third higher than those treated by obstetricians trained in the better-performing programs.⁴⁰ Other linkage work with a billing (cost) focus revealed physician spending patterns were associated with regional spending patterns during their residency training.⁴¹ The term ‘imprinting’ has been used for these long-lasting training effects.⁴⁰

Although graduates spend more time in training programs than they do in medical school, a recent US study⁴² linked a physician dataset (877 000 practitioners) with Medicare procedures and payment and showed geographic distribution of practitioners and their billing practices correlated with medical school tuition costs. The authors of that study also found other divergences suggesting ‘early influences during medical school may have lasting impacts on a physician’s future clinical decisions’.⁴² To understand more, a snapshot study such as this needs to be developed with longitudinal work. See Box 3 for more details.

The barriers to establishing the kinds of long-term follow-up of medical students we have described would be significant. These barriers exist in other countries and some have gained stakeholder cooperation for national data collections, or support for research projects in these areas.

Data of the kind we describe will enable prioritisation of improvements to our costly medical programs. These improvements are likely to include changes to entry processes, management of conduct and progression and balance of curriculum content (e.g. communication skills and health information literacy skills to support evidence-based practice). However, overemphasis on preparedness for the first year of work should be avoided, because this may risk biasing curriculum towards the instrumental skills required of a junior doctor with a negative effect on the development of skills for self-directed learning that will enable them to remain continuously competent in evidence-based care throughout their career. Ideally, a spiral curriculum⁴³ would follow the students into the workplace (the prevocational years) and even influence advanced training.

Such improvements to medical schooling become ‘preventative measures’ to ensure that future practitioners have better well being, there are fewer patient complaints and that a more reliable standard of evidence-based care is practised.

The authors declare no competing interests.