Register      Login
Journal of Primary Health Care Journal of Primary Health Care Society
Journal of The Royal New Zealand College of General Practitioners
RESEARCH ARTICLE (Open Access)

Using the Multiple Mini Interview for selection into vocational general practice training

Steven Lillis 1 2 , Liza Lack 1 , Allan Mbita 1 , Melissa Ashford 1
+ Author Affiliations
- Author Affiliations

1 Royal New Zealand College of General Practitioners, Level 4, 50 Customhouse Quay, Wellington 6143, New Zealand.

2 Corresponding author. Email: steven.lillis@outlook.co.nz

Journal of Primary Health Care 11(1) 75-79 https://doi.org/10.1071/HC18085
Published: 3 April 2019

Journal Compilation © Royal New Zealand College of General Practitioners 2019.
This is an open access article licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Abstract

INTRODUCTION: Interviews for selection into post graduate training courses are an accepted method of selection. There is the presumption that such interview processes are fair to both candidates and the training scheme.

AIM: Due to concerns over unconscious bias and a desire to move to best practice, the Royal New Zealand College of General Practitioners introduced the Mini Multiple Interview (MMI) process as the selection process for doctors wishing to enter vocational training in general practice.

METHODS: Aspects assessed during the interviews were developed through wide consultation and included: reason for wanting to undertake training, managing poor compliance, addressing issues of equity, managing complaints, insight and understanding the role of general practitioners in chronic care. There were 218 applicants who took the MMI. Demographic data as well as scores were collected.

RESULTS: The MMI process has good reliability and performs well in several aspects of validity. All three interview venues had similar results. There was no gender difference in overall result or scores. New Zealand graduates scored higher than overseas graduates. Of the 218 candidates, 12 were considered not yet ready to enter training.

DISCUSSION: The MMI process appears to have acceptable reliability and good validity. The structure of the MMI is likely to have reduced unconscious bias. Further research will study the predictive validity of the MMI for this cohort of candidates.

KEYWORDS: General practice; vocational training; interview.

Introduction

Admission processes for postgraduate training aim to select candidates who are likely to successfully complete the training and become competent practitioners in their discipline. A second objective is to design and implement a process that reflects the needs of a high-stakes assessment. Historically, interviews have been used as part of the process of candidate selection, with the typical structure being an in-depth single interview with two or three people. Concerns over issues of interrater reliability (would the interview score be the same regardless of who was interviewing and marking a candidate?), validity (does the test measure what it is supposed to measure?) and poor ability to predict future academic performance led to the development of alternative methods of candidate selection, such as the Multiple Mini Interview (MMI) for both medical school and vocational training programmes.13

The MMI is conceptually similar to an Objective Structured Clinical Examination (OSCE) with multiple stations, each assessing a different attribute and with a separate assessor for each station. Interviewees rotate through each station in turn and each interviewee is asked identical questions. The objective of the MMI is to assess ‘non-cognitive’ attributes of candidates.4 While traditional interviews are also used to measure these non-cognitive attributes, the MMI does so in a different way.5 Both situational questions such as ‘what would you do in this situation?’ and traditional interview questions: ‘tell me about yourself’ or ‘describe your strengths and weaknesses’ are used in the MMI.6 In general, between 7 and 12 stations are used, as this number gives an acceptable index of reliability.7 It is usual for a single assessor to mark a station. One study demonstrated acceptable reliability using six stations.8 An important feature of MMIs is the ability of individual education institutions to design a unique MMI where the content within the MMI structure reflects local imperatives.9 While such a bespoke assessment is of value to an institution, it also makes comparison between institutions difficult. As expected, the MMI shows little correlation with indices of cognitive function such as grade point averages.10 Although good content validity implies sampling across the entire breadth of attributes required for competent general practice, the MMI focuses on a subset of attributes that are not primarily cognitive. Further complicating the concept of content validity is that issues such as the context of general practice require some knowledge upon which to apply values and attitudes. There are now published guidelines on constructing an MMI.11

Research has shown that MMI is a viable method of selecting applicants to medical school and into vocational training. Of particular interest to New Zealand are studies that demonstrate better performance than traditional interviews in avoiding unintentional bias that could disadvantage students on socioeconomic or cultural grounds.12 The MMI is not immune to individual bias, but interviewer training may ameliorate this.13

Several studies have assessed the ability of the MMI to predict future performance. One study suggested that the MMI can predict future humanistic attributes but not performance on cognitive tests such as Multiple Choice Questions at undergraduate level.14 Recent research found more global correlations, including with knowledge tests.15 In postgraduate general practice vocational training, the MMI predicted scores on all assessments, including assessments usually associated with cognitive ability.16

The Royal New Zealand College of General Practitioners (RNZCGP) decided to use the MMI as a replacement for the traditional interview format and the applicants interviewed for the 2018 intake of registrars were assessed using this technique. The reason behind the initiative was concern over unconscious bias by the interviewers. The previous interview process used two or three interviewers who worked through a checklist of questions during an interview that lasted between 30 and 45 min. A co-design team was formed to steer the MMI initiative. Questions for the interviews were developed and then tested using an iterative process with a group consisting of medical educators and trainees. A marking scheme was then developed with a possible total of 12 marks for each question. The questions were refined by the examinations team into aspects of candidates that the College considered important. Each of the six aspects was measured in three ways, with four marks for each measure, giving a total of 72 available marks. These aspects are shown in Table 1.


Table 1. Interview questions and marking structure
Click to zoom

Interviewer training was developed and applied to reduce inter-rater variability. Interviewer training started with video recordings of mock interviews. Professional actors then played the role of candidates. A decision was made to use one examiner per station and to maintain the tradition of not charging candidates for the interview. This paper reports the results of the process for the first cohort of applicants to undergo the MMI.


Methods

Scores for all candidates were collected. Candidates’ demographic data and interview location were also collected. We used SPSS (version 25, SPSS Inc., IBM Corp., Armonk, NY, USA) for data analysis. Scores for each aspect are a composite score of the three measures in Table 1. We used the chi-square test for regional differences and differences between genders. Cronbach’s Alpha (α; a statistical measure of the reliability of a test) was calculated as an index of the reliability of the interview process, across all six aspects of the interview and all 218 candidates. We then recalculated the α with each item in turn deleted, to estimate which items contributed more to the overall α. Student’s t-tests were used to compare scores by source of primary medical qualification and by gender.


Results

Results by venue

A total of 217 candidates attended the interview process in Auckland, Wellington and Christchurch. Numbers interviewed in each region and success rates for gaining a training position are given in Table 2. There was no significant difference in the pass rates by region (P = 0.17).


Table 2. Success rates per region
T2

Reliability

Across all six aspects of the interview and all 218 candidates, the α coefficient was 0.735. Aspects 1 (α = 0.679) and 3 (α = 0.672) contributed most to the overall reliability; aspects 4 (α = 0.719) and 5 (α = 0.716) the least.

Results by gender

Gender analysis revealed more female (122) than male (95) candidates presenting for interview. There was no significant gender difference in success rates (95.8% of males and 93.4% of females were successful; P > 0.05); however, there were some differences in the score distribution of male and female candidates. The mean score for female candidates was slightly higher than for male candidates (53.8 for females and 51.8 for males), but this difference was not statistically significant (P = 0.059).

Candidate performance on aspects

Each of the six MMI aspects was marked separately. Overall, the mean scores were similar for each of the six aspects and were also similar for males and female candidates except for aspect 1, choice of GP training, where the mean score for males was significantly lower than for females (males 7.8, females 8.8; P = 0.035).

Characteristics of unsuccessful candidates

A comparison of successful and unsuccessful candidate showed that aspects 2 (relating to patient non-compliance), 3 (relating to Māori health) and 5 (relating to candidates’ self-awareness) contributed most to unsuccessful scores.

Results by source of primary medical qualification

Demographic data from candidates were used to compare scores from graduates of Auckland, Otago and Other medical education programmes (Table 3). We found that New Zealand graduates’ scores were significantly higher than scores of overseas graduates (mean for New Zealand graduates = 54.7 vs. 49.6 for Other graduates; P = 0.001).


Table 3. Mean score of candidates by source of primary medical qualification
T3

Comparison of candidates on each aspect by source of primary medical qualification was undertaken. New Zealand graduates (universities of Auckland and Otago) were compared to all other graduates. The analysis shows that all aspects had significant differences between these two groups. These data are presented in Table 3.


Discussion

Van der Vleuten’s model for assessing assessments encompasses the elements of reliability, validity, feasibility, acceptability and educational effect.17 Reliability was measured using Cronbach’s α,18 with the α coefficient of 0.735 indicating acceptable reliability for an assessment such as this, with only six stations. Increasing the reliability could be achieved by adding further stations, but this would come at extra cost. The analysis of ‘α with item deleted’ revealed that aspects 4 (managing a patient complaint) and 5 (personal insight) contributed least to the overall α.

The validity of the MMI process is a complex consideration. Content validity with regard to non-cognitive attributes was assured by the process of developing the interview items. An important objective of content validity is to design a process that did not select on the basis of gender. This analysis is reassuring about the content validity of the MMI when comparing data from the two New Zealand universities and candidates from all other universities, but the analysis did find that graduates from New Zealand universities scored significantly better than other candidates. An obvious explanation for this finding is that New Zealand graduates have a better preparation from their undergraduate training for the non-cognitive skills and knowledge that is tested in the MMI. An alternative explanation is that there was systemic bias in the construction of the questions, systemic bias in marking of candidates or both. Given that the structure of the process, the content of the questions and the training of the interviewers was specifically designed to reduce bias, it is more likely that New Zealand undergraduate training is better preparation for the interviews than overseas training.

While it may have been useful to analyse the data using both socioeconomic and cultural background, these variables can have multidimensional constructs, making statistical analysis of such data problematic. Concurrent validity is the degree to which the test correlates with other measures of the same construct that are measured at the same time. It is reassuring that the scores of candidates showed no significant difference between venues, as this reflects good concurrent validity of this MMI process. The MMI does require some extra resourcing beyond the traditional interview method; however, these are not problematic, and the overall cost is only marginally greater.

The MMI process has several potential educational effects. In the future and as further data from this cohort become available, consequential (predictive) validity can be assessed when results from formal assessments are available for successful candidates. This will allow understanding of the predictive value of the interview process on a range of variables, both cognitive and non-cognitive. This report on the MMI process will aid future candidates to better understand the process and the psychometrics supporting it. Further, it will inform the RNZCGP on the utility of the MMI process for selecting candidates using defined criteria.


Conclusion

The MMI process as designed and used by the RNZCGP for candidate selection for entry to vocational general practice training was used for the first time to select candidates for the 2018 entry. The process displays acceptable indices of reliability. On several aspects of validity, the results are reassuring. The new selection process should select candidates who are better suited to general practice training than the previous process. MMI also allows psychometric analysis, a feature that was difficult to undertake with traditional interview techniques. This enables an iterative approach to quality improvement. It may also lead to better understanding of a particular candidate’s strengths and weaknesses in non-cognitive skills that could be reflected in the education offered during registrar training. Further research is planned that will study the predictive validity of the MMI on both cognitive and non-cognitive assessments that will be undertaken later in the general practice training programme for this cohort.


FUNDING

This research did not receive any specific funding.


COMPETING INTERESTS

The authors have no conflicts of interest.



References

[1]  Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ. 2004; 38 314–26.
An admissions OSCE: the multiple mini-interview.Crossref | GoogleScholarGoogle Scholar | 14996341PubMed |

[2]  Morris JG. The value and role of the interview in the student admissions process: a review. Med Teach. 1999; 21 473–81.
The value and role of the interview in the student admissions process: a review.Crossref | GoogleScholarGoogle Scholar |

[3]  Goho J, Blackman A. The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teach. 2006; 28 335–40.
The effectiveness of academic admission interviews: an exploratory meta-analysis.Crossref | GoogleScholarGoogle Scholar | 16807172PubMed |

[4]  Griffin B, Wilson I. Associations between the big five personality factors and multiple mini-interviews. Adv Health Sci Educ Theory Pract. 2012; 17 377–88.
Associations between the big five personality factors and multiple mini-interviews.Crossref | GoogleScholarGoogle Scholar | 21751103PubMed |

[5]  Griffin B, Auton J, Duvivier R, et al. Multiple mini interviews: revealing similarities across institutions. BMC Med Educ. 2018; 18 190
Multiple mini interviews: revealing similarities across institutions.Crossref | GoogleScholarGoogle Scholar | 30081893PubMed |

[6]  Yoshimura H, Kitazono H, Fujitani S, et al. Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliability and acceptability comparison. BMC Med Educ. 2015; 15 75
Past-behavioural versus situational questions in a postgraduate admissions multiple mini-interview: a reliability and acceptability comparison.Crossref | GoogleScholarGoogle Scholar | 25890189PubMed |

[7]  Rees EL, Hawarden AW, Dent G, et al. Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37. Med Teach. 2016; 38 443–55.
Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37.Crossref | GoogleScholarGoogle Scholar | 27050026PubMed |

[8]  Yamada T, Sato J, Yoshimura H, et al. Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission. BMC Med Educ. 2017; 17 57
Reliability and acceptability of six station multiple mini-interviews: past-behavioural versus situational questions in postgraduate medical admission.Crossref | GoogleScholarGoogle Scholar | 28302124PubMed |

[9]  Reiter H, Eva K. Vive la Différence: the freedom and inherent responsibilities when designing and implementing multiple mini-interviews. Acad Med. 2018; 93 969–71.
Vive la Différence: the freedom and inherent responsibilities when designing and implementing multiple mini-interviews.Crossref | GoogleScholarGoogle Scholar | 29095171PubMed |

[10]  Pau A, Chen YS, Lee VK, et al. What does the multiple mini interview have to offer over the panel interview? Med Educ Online. 2016; 21 29874
| 26873337PubMed |

[11]  Eva KW, Macala C, Fleming B. Twelve tips for constructing a multiple mini-interview. Med Teach. 2018;
Twelve tips for constructing a multiple mini-interview.Crossref | GoogleScholarGoogle Scholar | 30299204PubMed |

[12]  Taylor CA, Green KE, Spruce A. Evaluation of the effect of socio-economic status on performance in a Multiple Mini Interview for admission to medical school. Med Teach. 2015; 37 59–63.
Evaluation of the effect of socio-economic status on performance in a Multiple Mini Interview for admission to medical school.Crossref | GoogleScholarGoogle Scholar | 24986755PubMed |

[13]  Roberts C, Clark T, Burgess A, et al. The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training. BMC Med Educ. 2014; 14 169
The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training.Crossref | GoogleScholarGoogle Scholar | 25123968PubMed |

[14]  Terregino CA, Copeland HL, Laumbach SG, et al. How good are we at selecting students that meet our mission? Outcomes of the 2011 and 2012 entering classes selected by a locally developed multiple mini interview. Med Teach. 2018; 40 1300–5.

[15]  Jerant A, Henderson MC, Griffin E, et al. Do admissions multiple mini-interview and traditional interview scores predict subsequent academic performance? A study of five California medical schools. Acad Med. 2019; 94 388–95.
Do admissions multiple mini-interview and traditional interview scores predict subsequent academic performance? A study of five California medical schools.Crossref | GoogleScholarGoogle Scholar | 30188370PubMed |

[16]  Patterson F, Rowett E, Hale R, et al. The predictive validity of a situational judgement test and multiple-mini interview for entry into postgraduate training in Australia. BMC Med Educ. 2016; 16 87
The predictive validity of a situational judgement test and multiple-mini interview for entry into postgraduate training in Australia.Crossref | GoogleScholarGoogle Scholar | 26957002PubMed |

[17]  Van Der Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996; 1 41–67.
The assessment of professional competence: developments, research and practical implications.Crossref | GoogleScholarGoogle Scholar |

[18]  Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011; 2 53–5.
Making sense of Cronbach’s alpha.Crossref | GoogleScholarGoogle Scholar | 28029643PubMed |