Exploring how a patient encounter tracking and learning tool is used within general practice training: a qualitative study

Study aim

This study aimed to provide greater understanding of how GP registrars, MEs, and supervisors use a PETAL tool within Australian general practice training, using a case study of the longitudinal Registrars’ Clinical Encounters in Training (ReCEnT) project. The study builds on quantitative survey findings regarding perceptions of registrars, MEs, and supervisors about the utility of ReCEnT for reflection on, and change to, registrar learning and clinical practice.¹⁴

Context

In 2020–21, ReCEnT was used in three regional training organisations (RTOs), accounting for 44% of all registrars in Australian GP training.¹⁸ ReCEnT, as a formative educational tool in use since 2010, is designed to assist GP registrars, along with their supervisors and MEs, to reflect on their practice, on their educational needs, and to encourage quality improvement.⁴ In brief, once in each of their three 6-month mandatory general practice training terms, registrars complete details about 60 consecutive consultations, documenting information about themselves, their patients and the consultations (including registrars’ clinical actions during the consultations). In each term, registrars receive an individualised feedback report summarising this information. Each report provides comparisons of registrars’ results with their own results over time (ie term-to-term), with aggregate registrar data, and with previously published national data for established GPs, where available.¹⁹ Prompts within the report ask registrars to critically reflect on the findings presented – particularly considering: how typical the consultations were of their usual practice; and how their practice’s demographics, policies, and procedures might have influenced the findings. Reports are delivered within 3 weeks of data being provided by registrars to facilitate timely reflection.^20–22 The ReCEnT project is described more fully elsewhere.²³ In 2020, ReCEnT was included in a pilot programmatic assessment program, implemented in one RTO. Thus, questions about ReCEnT’s potential role in programmatic assessment were included in this research.

Study design

An interpretivist perspective was chosen as an appropriate theoretical framework²⁴ to understand and explain the phenomenon of interest, namely perceptions of how ReCEnT as a PETAL tool is used for learning and for use in programmatic assessment. Initially, focus groups, with key informants who were drivers of the use of ReCEnT within GP registrar education and with knowledge of the programmatic assessment pilot, were used to identify pre-existing assumptions and to inform interview guides (Supplementary File S1) for a broader sample of registrars, MEs, and supervisors.

Ethics approval was obtained from the University of Newcastle Human Research Ethics Committee (approval number H-2020-0103).

Participants and procedure

Participants were from two Australian RTOs: GP Synergy (covering NSW and ACT) and General Practice Training Tasmania. Inclusion criteria for participation were: 2020 GP registrars who had completed two or more rounds of ReCEnT and had completed their final round of ReCEnT (in General Practice Term 3) before the onset of the coronavirus disease 2019 (COVID-19) pandemic; and MEs and supervisors who had a registrar(s) complete ReCEnT in 2019. Key informants for focus groups were identified by the two RTOs and directly invited. For interviews, three avenues of invitation were deployed: (1) key informants who could not attend a focus group (eg unsuitable time) were automatically invited; (2) participants who completed a quantitative survey (conducted as part of a larger study)¹⁴ were invited to express interest in an interview; and (3) an email invitation was sent by each RTO to further seek interest in interviews by registrars, supervisors and MEs who met the inclusion criteria but did not respond to the quantitative survey. Among those who expressed an interest (avenues 2 and 3), purposive selection was undertaken to get a broad sample based on demographic variables (eg age, gender, country of primary medical degree, location of practice). Participants gave signed consent to take part in the study and received a AU$50 gift voucher for their time (except MEs who were employees of the RTO). The study’s chief investigator (LK) conducted the focus groups, and the Senior Qualitative Researcher (JT) conducted the interviews. Data collection was via video conference or phone. Interviews continued until no new ideas were shared.

Analysis and reflexivity

Qualitative data collection and analysis was concurrent and iterative, with the interview guide modified as needed in response to findings. NVivo 12 (QSR International) was used to organise and model the data. Inductive reflexive thematic analysis²⁵ was conducted by two researchers (JT, LK), employing a process of constant comparison. Comparative parallel coding resulted in an initial coding framework, then applied to transcripts and regularly modified during the iterative concurrent data collection/analysis process. First-order codes were organised into second-order themes. Codes and themes were collated then abstracted to form a theoretical description.

Analysis meetings were held regularly with all authors, to discuss coding and interpretation. Research team members were from both RTOs, had health professional backgrounds (eg medicine, psychology) and expertise in GP-related qualitative research, clinical practice, and medical education. Some members were involved in day-to-day ReCEnT project management, so provided insights from knowledge of the tool. The interviewers/analysts (JT, LK) were independent. During analysis and interpretation, a process of reflexivity was used, addressing each investigator’s pre-existing assumptions, experiences, and personal interests in the study.^26,27

Utility of registrars’ ReCEnT reports

Three themes centred on how ReCEnT reports were used following distribution to GP registrars, their supervisors and MEs. These themes were how ReCEnT reports: reassured registrars; facilitated reflection; and helped to identify registrar learning needs.

Reassuring registrars

Participants reported that ReCEnT feedback reports provided ‘interesting’ information about how registrars were tracking compared with previous terms, their peers, and established GPs. This was important to registrars, who often used the comparisons for reassurance that their practice and experience was ‘normal.’

I think it was more reassurance that I’m not doing the wrong thing. (Registrar, R10)

There was a caveat that ReCEnT data did not consider registrars’ individual circumstances or location (eg differences between urban and rural practices). For example, a rural/remote-located registrar argued that their ReCEnT report did not compare well with the ‘average’ registrar. Further, comparisons could potentially distress registrars who differed substantially from their peers.

What I do remember was that she [registrar] felt a bit distressed by the results because she found she was quite an outlier. (Supervisor, S12)

Facilitating reflection

Many participants, particularly registrars and MEs, described how ReCEnT reports facilitated registrars’ self-reflection. Sections of the report covered many different aspects of practice (eg patients seen including diagnoses/problems, registrars’ management actions and in-consultation assistance- and information-seeking), each providing opportunities to reflect.

…it’s a good reflective process for the registrar, and even for us to think about the patients that they’re seeing and try and teach them in areas that they might not be seeing as much of. (Supervisor, S15)

However, a few educators noted that some registrars might lack experience to self-reflect on their own without assistance from their ME or supervisor.

It’s really just about prodding them to think that way and to raise their awareness that there is that form of reflection to learn…reflective learning. (ME, M01)

In addition to the reports, in-consultation entering of ReCEnT data gave registrars opportunities to immediately reflect on their practice and their patient management options.

While I was filling it out, I found, to reflect on what I’d just done and sometimes it made me practice better! (Registrar, R01)

Identifying learning needs

ReCEnT feedback reports had utility in understanding and identifying registrars’ learning needs. Most MEs reported that ReCEnT was a valuable tool to start a conversation, to help identify learning needs, or to flag potential problems in registrars’ training experiences. MEs reported they used ReCEnT reports to assist with registrar remediation.

…it helps me with people that need remediation. People [registrars] that are progressing well, I don’t find it that useful. It confirms that they’re seeing a broad range of people [patients]. When somebody shows a deficit then I go looking with a fine-tooth comb back through their ReCEnT reports. (ME, M12)

Reviewing ReCEnT reports helped registrars, MEs and supervisors to address any learning or clinical gaps in registrars’ training experiences. Both MEs and supervisors played a significant role. Supervisors could provide immediate teaching to fill gaps in registrars’ clinical experience, whereas MEs could take a broader perspective, such as discussing choice of future practices, learning goals, and exam preparation.

I use it as part of the foundations for them to set up their training and learning plan, and to a degree, training advisory to help see where they’re at and what they’re seeing and what they’re not seeing. (ME, M08)

There were mixed views on whether reflecting on ReCEnT reports helped registrars change their clinical practice. Although some registrars reported change following reflection, others reported that ReCEnT did not influence change or that change was influenced by multiple factors and not ReCEnT alone.

…It was a good reflection point, but I didn’t necessarily change my practice based just on that [ReCEnT report]. I didn’t change my practice because I didn’t want to change my practice. I feel more comfortable when I’m being thorough… (Registrar, R15)

MEs and supervisors reported that they often did not know whether a registrar had implemented change in response to ReCEnT feedback. This often occurred because registrars regularly changed practice location and supervisors each term. Also, supervisors, in particular, reported that their registrar did not need to change.

I think it [ReCEnT] has the potential to give a registrar the information they need to make a change. I haven’t actually seen that because it hasn’t been applicable. There’s been no need to change for my registrars. (Supervisor, S16)

Enabling ReCEnT to reach its full potential

Three themes arose where participants shared their insights into key enablers that enhanced ReCEnT’s potential as a reflection and learning tool, specifically: where there is a culture of reflection; where meaningful discussions with supervisors and MEs occur; and where objective data are valued.

A culture of reflection

The culture of medical training was reported as slowly changing to value reflection more, and that this change was progressing from the university system through to GP training.

I went through [university] and that was a lot of reflective practice, and probably a lot of the local MEs went there as well…so I guess they might have more time to develop those reflective thoughts. (Registrar, R15)

MEs recognised this transition to becoming better reflective practitioners.

Registrars now, are more reflective than they were when I was a registrar, who are probably more reflective than the registrars who came years before them. I think that we are slowly fostering reflective practice more and more in our trainees, but it’s a slow process and some people are more amenable to that than others. (ME, M04)

Supervisors reported the importance of giving registrars extra time to enter ReCEnT data to make registrars feel comfortable and safe to reflect on their practice. MEs went further to suggest the importance of having a community of support for MEs and supervisors so that ideas could be shared about how to use feedback from ReCEnT to improve learning opportunities for registrars.

Meaningful discussions

Participants reported that ReCEnT was better utilised for reflection when meaningful discussions about feedback reports occurred between registrars and educators. Discussions were key because even when registrars had reflected on or noticed that they were outliers, they needed educators’ expertise to appreciate if differences were substantive in their circumstances, and then needed the expertise to make changes.

Because you don’t know what you don’t know, and you can’t reflect on something you don’t understand. So, if you don’t have someone helping you facilitate that reflection, I think that would make it difficult. (Registrar, R08)

In recognising the importance of discussing ReCEnT, some MEs reported contacting both registrars and supervisors to ensure ReCEnT was discussed.

We have a few meetings… just gauging what other supervisors are doing, let them speak about that. You can obviously see the cogs turning in other people’s heads, so it seems to be a useful way of finding out what other people in this community of practice are actually doing. (ME and Supervisor, M05)

Valuing objective data

Valuing objective data in ReCEnT, irrespective of whether it confirmed or disconfirmed what participants thought, was a factor in an ongoing engagement with ReCEnT.

I probably felt… that I was seeing a lot more [problems relating to] skin than I actually was. That was a bit surprising for me… The ReCEnT data is a more objective tool in assessing that. (Registrar, R05)

As supervisors, we get taught how to try and get our registrar to self-reflect. …. ReCEnT is a wonderful tool to help that process on, because there is no denying it…it’s on black and white there. (Supervisor, S11)

Some participants reported that ReCEnT feedback reports confirmed what they already knew but this still had value as objective data.

I do like finding out that what I think is happening is actually what’s happening. I do think it’s [ReCEnT] useful even if there are no surprises. (Supervisor, S16)

The longitudinal nature of ReCEnT was also appreciated for providing evidence of progress.

I think the more ReCEnTs you’ve done, the more valuable the reports have been in looking at progress (Registrar, R03)

ReCEnT’s role within a programmatic assessment framework

A programmatic approach to assessment was first introduced in the two participating RTOs in 2020. Although MEs knew of programmatic assessment, registrars and supervisors were less familiar with it. Consequently, participants were given an explanation about programmatic assessment as part of the interview. The ensuing discussion identified two themes about ReCEnT’s role in programmatic assessment: as a tool for learning; and as one piece of the puzzle.

ReCEnT as a tool for learning

Some MEs described ReCEnT as a suitable component of programmatic assessment because it provided feedback for learning, rather than assessment of learning.

So, if you are thinking about this [ReCEnT] as a formative assessment… you’re looking at the registrar’s self-reflection, their ability to self-reflect and put that towards their learning and improvement (ME, FGP05)

If we keep reminding ourselves that this is an assessment of the demographics that the registrars are being exposed to and what they’re actually seeing… That’s definitely giving something that not any other assessment gives us. (ME, FGP10)

In contrast, supervisors and registrars had trouble locating ReCEnT within a programmatic assessment framework because they viewed the word ‘assessment’ as only representing an assessment of learning, rather than an assessment for learning.

I think you have to choose whether it’s going to be a self-reflection or whether it’s going to be an assessment. I don’t think it could be both… (Supervisor, S13)

I think the ReCEnT data is more a learning tool. I don’t think it should or can be used for assessment purposes. (Registrar, R05)

Although maintaining a summative rather than a formative view of assessments, participants suggested additions to ReCEnT to meet perceived ‘assessment’ requirements, such as a reflective essay. However, the additional workload was acknowledged as impractical and potentially unhelpful.

…the idea [of ReCEnT] is great but the problem is formalising it becomes a paper exercise rather than actually being any help to people because all you end up is writing essays and stories. I’ve done hundreds of Alice in Wonderland stories about reflection while I was in [country] because that was what was expected. But that made me learn absolutely nothing whatsoever apart from how to write stories. (Supervisor, S01)

I don’t see how helpful it would be versus how much extra workload that would generate for both the registrars and whoever was going to have to review and mark that… (ME, M10)

‘One piece of the puzzle’ in programmatic assessment

A premise of programmatic assessment is to have multiple assessments for learning, across various time points in training, to develop the competencies to become a GP. ReCEnT was perceived as one source of information about registrars’ progress.

I use it [ReCEnT] as just one piece of the puzzle and all the other tools, are… adding in to get the global picture… I don’t think that ReCEnT alone, as a one-only chore is going to give you all the information you would like… (ME, MO8)

Main findings

This study further clarifies the utility of ReCEnT as a PETAL tool, as established in the quantitative arm of the project,¹⁴ and offers greater understanding of the potential of ReCEnT as a tool that not only helps to identify GP registrars’ learning needs, but also provides objective data on registrars’ clinical exposure. Although ReCEnT feedback reports aid in providing reassurance and promoting self-reflection by registrars on their practice, ReCEnT has greater potential where there is a culture of reflection leading to opportunities for meaningful discussions with supervisors and MEs. What is less clear is the perceived role of this PETAL tool within a Programmatic Assessment framework.

A structure–process–outcome model for using a PETAL tool in programmatic assessment

We propose an approach based on the synthesis of our results that draws upon Donabedian’s Structure–Process–Outcomes model.²⁸ The model has been used in Australian general practice.²⁹ Our approach is depicted in Fig. 2.

Fig. 2.

Using a PETAL tool in programmatic assessment: structure, process, outcome.

In Australia’s apprenticeship-style model of GP training, in-practice learning and experience is central to the development of confidence and clinical competencies.¹ With changes in case complexity,³⁰ it is imperative that GP training provides opportunities for a broad patient mix.^15,31 Clinical performance can manifest over a series of encounters³² as captured using a PETAL tool. However, using a PETAL tool as an assessment in GP training is problematic if maintaining a summative rather than formative view of assessment (our study participants tended to use ‘formative/summative’, rather than ‘low-stakes/high-stakes’ as used in the programmatic assessment literature).^7,33 Participants recognised that ReCEnT reports can provide feedback for learning, consistent with the view ‘that current research focuses on the ‘validity’ of the user and their way of interacting with the assessment instrument rather than purely the validity of the instrument’.⁷ Thus, a structure is needed to set up assessments for learning in GP training where there is effective engagement and reflection on registrars’ clinical exposure.^21,22,34,35

The process of tracking and learning using PETALs needs a positive culture of feedback.^36,37 In this process, registrars are active participants in feedback, not just receivers of information.²² In this humanist approach to learning, supervisors and MEs have roles as coaches²² and facilitators of meaningful discussions.^21,37 Our findings suggest that ReCEnT’s perceived value and utility arise from how registrars and their educators interact with it. It is in the process of engagement and reflection that learning occurs. Interestingly, many supervisors commented that their registrars performed ‘as expected’, yet did not use ReCEnT’s extensive feedback to extend their registrars even if performing well.

Finally, how can a PETAL, such as ReCEnT, fit into a framework of programmatic assessment of GP training?¹⁰ First, ReCEnT is one, among many, instruments/methods used for learning in the programmatic assessment of GP training.⁷ For PETALs to be effective in this framework, they need to be valued for what they provide, rather than being seen as onerous,³⁸ or perfunctory.³⁹ An advantage of ReCEnT as a longitudinal PETAL tool is that it provides multiple points across GP training for an assessment that can drive learning needs.⁸ Thus, registrars can use ReCEnT as a formative assessment of their continuity of experience, which would seem to fit well within a programmatic assessment framework.⁴⁰ For example, a decline in registrar’s in-consultation advice-seeking across successive training terms might be evidence of increasing confidence,⁴ which could be confirmed in discussion with educators.

The challenge remains to address disagreements or misunderstandings about assessment for learning and assessment of learning.⁴¹ Although we found ReCEnT, as a formative assessment, assists in understanding registrars’ clinical experiences, confirming its educational utility as an assessment tool mighty need to go beyond this.¹³ Proponents of programmatic assessment argue that there needs to be a summative or low-stakes component to ensure each registrar demonstrates agency by reflecting on the feedback and discussing this with their educators, for optimal learning to occur.⁷ We know from our quantitative study that reflection and discussion do not always occur.¹⁴ In this study, interviewees varied markedly on this issue – some insistent on low-stakes ‘consequences’, whereas others arguing any summative component would interfere with registrars’ honest recording of their behaviour. Clearly, more work needs to be done on ensuring that this PETAL tool is fit for purpose in a programmatic assessment framework. Donabedian’s Structure–Process–Outcomes model provides a basis for a safe and supportive approach to achieve this fit for purpose.^10,28

Strengths and weaknesses of the study

This study was conducted across two Australian RTOs who use ReCEnT. These two RTOs were responsible for training 36% of all Australian registrars in general practice terms and have a demographic and geographic presence across the range of Australian GP vocational training.⁴² There might have been different perspectives in other RTOs.

Engaging time-poor GPs is always a challenge.⁴³ Our strategy was to use multiple methods to recruit sufficient registrars, supervisors and MEs. Overall, we achieved a relatively even mix of registrars, supervisors and MEs; however, we acknowledge there might be a volunteer bias from participants with strong feelings on the topic.