Multicriteria decision analysis (MCDA) for health technology assessment: the Queensland Health experience
Sarah Howard A F , Ian A. Scott B , Hong Ju C , Liam McQueen A and Paul A. Scuffham D EA Healthcare Evaluation and Assessment of Technology, Healthcare Improvement Unit, Clinical Excellence Division, Queensland Department of Health, Level 2, 15 Butterfield Street, Herston, Qld 4006, Australia. Email: Liam.McQueen@health.qld.gov.au
B Department of Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Qld 4102, Australia. Email: ian_scott@health.qld.gov.au
C Agency for Care Effectiveness, Ministry of Health, 16 College Road, Singapore. Email: hongju04@yahoo.com
D Menzies Health Institute Queensland, Griffith University, Nathan, Brisbane, Qld 4111, Australia. Email: p.scuffham@griffith.edu.au
E Centre for Applied Health Economics, Griffith University, Nathan, Brisbane, Qld 4111, Australia.
F Corresponding author. Email: sarah.howard@health.qld.gov.au
Australian Health Review 43(5) 591-599 https://doi.org/10.1071/AH18042
Submitted: 1 March 2018 Accepted: 9 August 2018 Published: 12 September 2018
Abstract
Objectives In determining whether new health technologies should be funded, health technology assessment (HTA) committees prefer explicit to implicit methods of analysis in enhancing transparency and consistency of decision making. The aim of this study was to develop and pilot a multicriteria decision analysis (MCDA) framework for the Queensland Department of Health HTA program committee, which weighted decision making criteria according to their perceived importance as determined by group consensus.
Methods The criteria used in the MCDA framework were identified by reviewing the five unweighted criteria used in the existing process, consultation with committee members and literature review. Criteria were clearly defined and ordinal categories of lowest to highest preferred were assigned against which technology submissions would be rated. Criteria weights were determined through a discrete choice experiment (DCE) survey of committee members using validated software. Mean weighted technology scores were then used to guide deliberative discussions in determining final funding decisions.
Results The MCDA framework created one additional criterion to the previous five. The criteria and their mean weights identified through the DCE survey were clinical benefit and safety (27.2%), quality of evidence (19.2%), implementation capacity (16.9%), innovation (15.4%), burden of disease and clinical need (13.3%) and societal and ethical values (8.0%). Criterion weights varied considerably between individual committee members, with one criterion having a difference of 36.9% between the highest and lowest preference weights. Following deliberative discussions, all but one of 10 submissions were awarded funding. The submission not supported received the third lowest score through the MCDA model.
Conclusions This pilot application of an MCDA framework, as a complement to committee deliberation, conferred greater transparency and objectivity on HTA assessment of technologies. The framework converted an implicit, unweighted review process to one that is more explicit, flexible in weighting importance and pragmatic.
What is known about the topic? HTA programs involve complex decision-making processes requiring the consideration of multiple criteria. Explicit methods of analysis that use weighted criteria according to their relative importance enhance transparency and consistency of decision making by HTA committees, and are preferred to implicit reviews using unweighted criteria.
What does this paper add? This article describes the development and piloting of an MCDA framework that aims to improve transparency, objectivity and consistency of funding decisions of the Queensland HTA committee. Criteria were identified through a review of current processes, committee discussions and a literature review, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) quality of evidence system. Criteria were weighted using a discrete choice experiment involving committee members. Using weighted criteria, mean technology scores were calculated and incorporated into deliberative discussions to determine funding decisions.
What are the implications for practitioners? The MCDA framework described here converted a more implicit, unweighted process to one that was more pragmatic, explicit and flexible in scoring HTA submissions. This framework may be useful to other HTA programs and could be expanded to resource allocation decision making in many other healthcare settings.
References
[1] Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal. Med Decis Making 2012; 32 376–88.| Bridging health technology assessment (HTA) and efficient health care decision making with multicriteria decision analysis (MCDA): applying the EVIDEM framework to medicines appraisal.Crossref | GoogleScholarGoogle Scholar |
[2] Wahlster P, Goetghebeur MM, Schaller S, Kriza C, Kolominsky-Rabas P. Exploring the perspectives and preferences for HTA across German healthcare stakeholders using a multi-criteria assessment of a pulmonary heart sensor as a case study. Health Res Policy Syst 2015; 13 24
| Exploring the perspectives and preferences for HTA across German healthcare stakeholders using a multi-criteria assessment of a pulmonary heart sensor as a case study.Crossref | GoogleScholarGoogle Scholar |
[3] Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care 2007; 23 310–15.
| Priority setting for health technology assessments: a systematic review of current practical approaches.Crossref | GoogleScholarGoogle Scholar |
[4] Poder TG. Using the health technology assessment toolbox to facilitate procurement: the case of smart pumps in a Canadian hospital. Int J Technol Assess Health Care 2017; 33 54–62.
| Using the health technology assessment toolbox to facilitate procurement: the case of smart pumps in a Canadian hospital.Crossref | GoogleScholarGoogle Scholar |
[5] International Network of Agencies for Health Technology Assessment. What is health technology assessment (HTA)? 2017. Available at: http://www.inahta.org/ [verified 5 October 2017].
[6] Tanios N, Wagner M, Tony M, Baltussen R, van Til J, Rindress D, Kind P, Goetghebeur MM. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care 2013; 29 456–65.
| Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers.Crossref | GoogleScholarGoogle Scholar |
[7] Tsiachristas A, Cramm JM, Nieboer A, Rutten-van Mölken M. Broader economic evaluation of disease management programs using multi-criteria decision analysis. Int J Technol Assess Health Care 2013; 29 301–8.
| Broader economic evaluation of disease management programs using multi-criteria decision analysis.Crossref | GoogleScholarGoogle Scholar |
[8] Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc 2006; 4 14
| Priority setting of health interventions: the need for multi-criteria decision analysis.Crossref | GoogleScholarGoogle Scholar |
[9] Abrishami P, Oortwijn W, Hofmann B. Ethics in HTA: examining the ‘need for expansion’. Int J Health Policy Manag 2017; 6 551–3.
| Ethics in HTA: examining the ‘need for expansion’.Crossref | GoogleScholarGoogle Scholar |
[10] Trueman P, Grainger DL, Downs KE. Coverage with evidence development: applications and issues. Int J Technol Assess Health Care 2010; 26 79–85.
| Coverage with evidence development: applications and issues.Crossref | GoogleScholarGoogle Scholar |
[11] Schey C, Krabbe PFM, Postma MJ, Connolly MP. Multi-criteria decision analysis (MCDA): testing a proposed MCDA framework for orphan drugs. Orphanet J Rare Dis 2017; 12 10
| Multi-criteria decision analysis (MCDA): testing a proposed MCDA framework for orphan drugs.Crossref | GoogleScholarGoogle Scholar |
[12] Bridges JFP, Hauber AB, Marshall D, Lloyd A, Prosser LA, Regier DA, Johnson FR, Mauskopf J. Conjoint analysis applications in health – a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health 2011; 14 403–13.
| Conjoint analysis applications in health – a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force.Crossref | GoogleScholarGoogle Scholar |
[13] Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health 2012; 15 1172–81.
| Multiple criteria decision analysis for health technology assessment.Crossref | GoogleScholarGoogle Scholar |
[14] Devlin N, Sussex J. Incorporating multiple criteria in HTA: methods and processes. London: Office of Health Economics; 2011.
[15] Marsh K, IJzerman M, Thokala P, Baltussen R, Boysen M, Kaló Z, Lönngren T, Mussen F, Peacock S, Watkins J, Devlin N. Multiple criteria decision analysis for health care decision making – emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 2016; 19 125–37.
| Multiple criteria decision analysis for health care decision making – emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force.Crossref | GoogleScholarGoogle Scholar |
[16] Ju H, Hewson K. Health technology assessment and evidence-based policy making: Queensland Department of Health experience. Int J Technol Assess Health Care 2014; 30 595–600.
| Health technology assessment and evidence-based policy making: Queensland Department of Health experience.Crossref | GoogleScholarGoogle Scholar |
[17] The Ontario Health Technology Advisory Committee. Decision determinants guidance document. 2010. Available at http://www.hqontario.ca/en/mas/tech/pdfs/2011/guide_decision.pdf [verified 27 August 2018]
[18] Sullivan T, Hansen P. Determining criteria and weights for prioritizing health technologies based on the preferences of the general population: a New Zealand pilot study. Value Health 2017; 20 679–86.
| Determining criteria and weights for prioritizing health technologies based on the preferences of the general population: a New Zealand pilot study.Crossref | GoogleScholarGoogle Scholar |
[19] Brożek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW, Phillips B, Lelgemann M, Lethaby A, Bousquet J, Guyatt GH, Schünemann HJ. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy 2009; 64 669–77.
| Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions.Crossref | GoogleScholarGoogle Scholar |
[20] Broekhuizen H, Groothuis-Oudshoorn CGM, van Til JA, Hummel JM, IJzerman MJ. A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. Pharmacoeconomics 2015; 33 445–55.
| A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions.Crossref | GoogleScholarGoogle Scholar |
[21] Brożek JL, Akl EA, Compalati E, Kreis J, Terracciano L, Fiocchi A, Ueffing E, Andrews J, Alonso-Coello P, Meerpohl JJ, Lang DM, Jaeschke R, Williams JW Jr, Phillips B, Lethaby A, Bossuyt P, Glasziou P, Helfand M, Watine J, Afilalo M, Welch V, Montedori A, Abraha I, Horvath AR, Bousquet J, Guyatt GH, Schünemann HJ. Grading quality of evidence and strength of recommendations in clinical practice guidelines part 3 of 3. The GRADE approach to developing recommendations. Allergy 2011; 66 588–95.
| Grading quality of evidence and strength of recommendations in clinical practice guidelines part 3 of 3. The GRADE approach to developing recommendations.Crossref | GoogleScholarGoogle Scholar |
[22] Marsh K, Dolan P, Kempster J, Lugon M. Prioritizing investments in public health: a multi-criteria decision analysis. J Public Health (Oxf) 2013; 35 460–6.
| Prioritizing investments in public health: a multi-criteria decision analysis.Crossref | GoogleScholarGoogle Scholar |
[23] Hansen P, Ombler F. A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives. J Multi-Crit Decis Anal 2008; 15 87–107.
| A new method for scoring additive multi-attribute value models using pairwise rankings of alternatives.Crossref | GoogleScholarGoogle Scholar |
[24] Thokala P. Multiple criteria decision analysis for health technology assessment. Sheffield: Decision Support Unit, School of Health and Related Research; 2011. Available at http://www.redcriteria.org/wp-content/uploads/2015/07/MCDA-for-HTA-DSU.pdf [verified 28 August 2018]
[25] Husereau D, Boucher M, Noorani H. Priority setting for health technology assessment at CADTH. Int J Technol Assess Health Care 2010; 26 341–7.
| Priority setting for health technology assessment at CADTH.Crossref | GoogleScholarGoogle Scholar |
[26] Eichler HG, Kong SX, Gerth WC, Mavros P, Jönsson B. Use of cost-effectiveness analysis in health-care resource allocation decision-making: how are cost-effectiveness thresholds expected to emerge?. Value Health 2004; 7 518–28.
| Use of cost-effectiveness analysis in health-care resource allocation decision-making: how are cost-effectiveness thresholds expected to emerge?.Crossref | GoogleScholarGoogle Scholar |
[27] Ritrovato M, Faggiano FC, Tedesco G, Derrico P. Decision-oriented health technology assessment: one step forward in supporting the decision-making process in hospitals. Value Health 2015; 18 505–11.
| Decision-oriented health technology assessment: one step forward in supporting the decision-making process in hospitals.Crossref | GoogleScholarGoogle Scholar |
[28] Oortwijn W, Sampietro-Colom L, Habens F. Developments in value frameworks to inform the allocation of healthcare resources. Int J Technol Assess Health Care 2017; 33 323–9.
| Developments in value frameworks to inform the allocation of healthcare resources.Crossref | GoogleScholarGoogle Scholar |
[29] Tony M, Wagner M, Khoury H, Rindress D, Papastavros T, Oh P, Goetghebeur MM. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Serv Res 2011; 11 329
| Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada.Crossref | GoogleScholarGoogle Scholar |
[30] Littlejohns P, Sharma T, Jeong K. Social values and health priority setting in England: ‘values’ based decision making. J Health Organ Manag 2012; 26 363–73.
