Using the Multiple Mini Interview for selection into vocational general practice training

KEYWORDS: General practice; vocational training; interview.

Admission processes for postgraduate training aim to select candidates who are likely to successfully complete the training and become competent practitioners in their discipline. A second objective is to design and implement a process that reflects the needs of a high-stakes assessment. Historically, interviews have been used as part of the process of candidate selection, with the typical structure being an in-depth single interview with two or three people. Concerns over issues of interrater reliability (would the interview score be the same regardless of who was interviewing and marking a candidate?), validity (does the test measure what it is supposed to measure?) and poor ability to predict future academic performance led to the development of alternative methods of candidate selection, such as the Multiple Mini Interview (MMI) for both medical school and vocational training programmes.^1–3

The MMI is conceptually similar to an Objective Structured Clinical Examination (OSCE) with multiple stations, each assessing a different attribute and with a separate assessor for each station. Interviewees rotate through each station in turn and each interviewee is asked identical questions. The objective of the MMI is to assess ‘non-cognitive’ attributes of candidates.⁴ While traditional interviews are also used to measure these non-cognitive attributes, the MMI does so in a different way.⁵ Both situational questions such as ‘what would you do in this situation?’ and traditional interview questions: ‘tell me about yourself’ or ‘describe your strengths and weaknesses’ are used in the MMI.⁶ In general, between 7 and 12 stations are used, as this number gives an acceptable index of reliability.⁷ It is usual for a single assessor to mark a station. One study demonstrated acceptable reliability using six stations.⁸ An important feature of MMIs is the ability of individual education institutions to design a unique MMI where the content within the MMI structure reflects local imperatives.⁹ While such a bespoke assessment is of value to an institution, it also makes comparison between institutions difficult. As expected, the MMI shows little correlation with indices of cognitive function such as grade point averages.¹⁰ Although good content validity implies sampling across the entire breadth of attributes required for competent general practice, the MMI focuses on a subset of attributes that are not primarily cognitive. Further complicating the concept of content validity is that issues such as the context of general practice require some knowledge upon which to apply values and attitudes. There are now published guidelines on constructing an MMI.¹¹

Research has shown that MMI is a viable method of selecting applicants to medical school and into vocational training. Of particular interest to New Zealand are studies that demonstrate better performance than traditional interviews in avoiding unintentional bias that could disadvantage students on socioeconomic or cultural grounds.¹² The MMI is not immune to individual bias, but interviewer training may ameliorate this.¹³

Several studies have assessed the ability of the MMI to predict future performance. One study suggested that the MMI can predict future humanistic attributes but not performance on cognitive tests such as Multiple Choice Questions at undergraduate level.¹⁴ Recent research found more global correlations, including with knowledge tests.¹⁵ In postgraduate general practice vocational training, the MMI predicted scores on all assessments, including assessments usually associated with cognitive ability.¹⁶

The Royal New Zealand College of General Practitioners (RNZCGP) decided to use the MMI as a replacement for the traditional interview format and the applicants interviewed for the 2018 intake of registrars were assessed using this technique. The reason behind the initiative was concern over unconscious bias by the interviewers. The previous interview process used two or three interviewers who worked through a checklist of questions during an interview that lasted between 30 and 45 min. A co-design team was formed to steer the MMI initiative. Questions for the interviews were developed and then tested using an iterative process with a group consisting of medical educators and trainees. A marking scheme was then developed with a possible total of 12 marks for each question. The questions were refined by the examinations team into aspects of candidates that the College considered important. Each of the six aspects was measured in three ways, with four marks for each measure, giving a total of 72 available marks. These aspects are shown in Table 1.

Table 1. Interview questions and marking structure

Interviewer training was developed and applied to reduce inter-rater variability. Interviewer training started with video recordings of mock interviews. Professional actors then played the role of candidates. A decision was made to use one examiner per station and to maintain the tradition of not charging candidates for the interview. This paper reports the results of the process for the first cohort of applicants to undergo the MMI.

Scores for all candidates were collected. Candidates’ demographic data and interview location were also collected. We used SPSS (version 25, SPSS Inc., IBM Corp., Armonk, NY, USA) for data analysis. Scores for each aspect are a composite score of the three measures in Table 1. We used the chi-square test for regional differences and differences between genders. Cronbach’s Alpha (α; a statistical measure of the reliability of a test) was calculated as an index of the reliability of the interview process, across all six aspects of the interview and all 218 candidates. We then recalculated the α with each item in turn deleted, to estimate which items contributed more to the overall α. Student’s t-tests were used to compare scores by source of primary medical qualification and by gender.

A total of 217 candidates attended the interview process in Auckland, Wellington and Christchurch. Numbers interviewed in each region and success rates for gaining a training position are given in Table 2. There was no significant difference in the pass rates by region (P = 0.17).

Table 2. Success rates per region

Across all six aspects of the interview and all 218 candidates, the α coefficient was 0.735. Aspects 1 (α = 0.679) and 3 (α = 0.672) contributed most to the overall reliability; aspects 4 (α = 0.719) and 5 (α = 0.716) the least.

Gender analysis revealed more female (122) than male (95) candidates presenting for interview. There was no significant gender difference in success rates (95.8% of males and 93.4% of females were successful; P > 0.05); however, there were some differences in the score distribution of male and female candidates. The mean score for female candidates was slightly higher than for male candidates (53.8 for females and 51.8 for males), but this difference was not statistically significant (P = 0.059).

Each of the six MMI aspects was marked separately. Overall, the mean scores were similar for each of the six aspects and were also similar for males and female candidates except for aspect 1, choice of GP training, where the mean score for males was significantly lower than for females (males 7.8, females 8.8; P = 0.035).

A comparison of successful and unsuccessful candidate showed that aspects 2 (relating to patient non-compliance), 3 (relating to Māori health) and 5 (relating to candidates’ self-awareness) contributed most to unsuccessful scores.

Demographic data from candidates were used to compare scores from graduates of Auckland, Otago and Other medical education programmes (Table 3). We found that New Zealand graduates’ scores were significantly higher than scores of overseas graduates (mean for New Zealand graduates = 54.7 vs. 49.6 for Other graduates; P = 0.001).

Table 3. Mean score of candidates by source of primary medical qualification

Comparison of candidates on each aspect by source of primary medical qualification was undertaken. New Zealand graduates (universities of Auckland and Otago) were compared to all other graduates. The analysis shows that all aspects had significant differences between these two groups. These data are presented in Table 3.

Van der Vleuten’s model for assessing assessments encompasses the elements of reliability, validity, feasibility, acceptability and educational effect.¹⁷ Reliability was measured using Cronbach’s α,¹⁸ with the α coefficient of 0.735 indicating acceptable reliability for an assessment such as this, with only six stations. Increasing the reliability could be achieved by adding further stations, but this would come at extra cost. The analysis of ‘α with item deleted’ revealed that aspects 4 (managing a patient complaint) and 5 (personal insight) contributed least to the overall α.

The validity of the MMI process is a complex consideration. Content validity with regard to non-cognitive attributes was assured by the process of developing the interview items. An important objective of content validity is to design a process that did not select on the basis of gender. This analysis is reassuring about the content validity of the MMI when comparing data from the two New Zealand universities and candidates from all other universities, but the analysis did find that graduates from New Zealand universities scored significantly better than other candidates. An obvious explanation for this finding is that New Zealand graduates have a better preparation from their undergraduate training for the non-cognitive skills and knowledge that is tested in the MMI. An alternative explanation is that there was systemic bias in the construction of the questions, systemic bias in marking of candidates or both. Given that the structure of the process, the content of the questions and the training of the interviewers was specifically designed to reduce bias, it is more likely that New Zealand undergraduate training is better preparation for the interviews than overseas training.

While it may have been useful to analyse the data using both socioeconomic and cultural background, these variables can have multidimensional constructs, making statistical analysis of such data problematic. Concurrent validity is the degree to which the test correlates with other measures of the same construct that are measured at the same time. It is reassuring that the scores of candidates showed no significant difference between venues, as this reflects good concurrent validity of this MMI process. The MMI does require some extra resourcing beyond the traditional interview method; however, these are not problematic, and the overall cost is only marginally greater.

The MMI process has several potential educational effects. In the future and as further data from this cohort become available, consequential (predictive) validity can be assessed when results from formal assessments are available for successful candidates. This will allow understanding of the predictive value of the interview process on a range of variables, both cognitive and non-cognitive. This report on the MMI process will aid future candidates to better understand the process and the psychometrics supporting it. Further, it will inform the RNZCGP on the utility of the MMI process for selecting candidates using defined criteria.

The MMI process as designed and used by the RNZCGP for candidate selection for entry to vocational general practice training was used for the first time to select candidates for the 2018 entry. The process displays acceptable indices of reliability. On several aspects of validity, the results are reassuring. The new selection process should select candidates who are better suited to general practice training than the previous process. MMI also allows psychometric analysis, a feature that was difficult to undertake with traditional interview techniques. This enables an iterative approach to quality improvement. It may also lead to better understanding of a particular candidate’s strengths and weaknesses in non-cognitive skills that could be reflected in the education offered during registrar training. Further research is planned that will study the predictive validity of the MMI on both cognitive and non-cognitive assessments that will be undertaken later in the general practice training programme for this cohort.

This research did not receive any specific funding.

The authors have no conflicts of interest.