Quality of condition suggestions and urgency advice provided by the Ada symptom assessment app evaluated with vignettes optimised for Australia*
Stephen Gilbert
A B C , Matthew Fenech A , Shubhanan Upadhyay A , Paul Wicks A and Claire Novorol A
+ Author Affiliations
- Author Affiliations
A Ada Health GmbH, Karl-Liebknecht-Straße 1, 10178 Berlin, Germany.
B EKFZ for Digital Health, University Hospital Carl Gustav Carus Dresden, Technische Universität Dresden, Dresden, Germany.
C Corresponding author. Email: science@ada.com
Australian Journal of Primary Health 27(5) 377-381 https://doi.org/10.1071/PY21032
Submitted: 22 February 2021 Accepted: 11 May 2021 Published: 14 October 2021
Journal Compilation © CSIRO 2021 Open Access CC BY-NC-ND
Abstract
When people face a health problem, they often first ask, ‘Is there an app for that?’. We investigated the quality of advice provided by the Ada symptom assessment application to address the question, ‘How do I know the app on my phone is safe and provides good advice?’. The app was tested with 48 independently created vignettes developed for a previous study, including 18 specifically developed for the Australian setting, using an independently developed methodology to evaluate the accuracy of condition suggestions and urgency advice. The correct condition was listed first in 65% of vignettes, and in the Top 3 results in 83% of vignettes. The urgency advice in the app exactly matched the gold standard 63% of vignettes. The app’s accuracy of condition suggestion and urgency advice is higher than that of the best-performing symptom assessment app reported in a previous study (61%, 77% and 52% for conditions suggested in the Top 1, Top 3 and exactly matching urgency advice respectively). These results are relevant to the application of symptom assessment in primary and community health, where medical quality and safety should determine app choice.
Keywords: artificial intelligence, clinical decision support, health app governance, patient-centred care, self-evaluation in healthcare, triage.
References
Ceney A, Tolond S, Glowinski A, Marks B, Swift S, Palser T (2020) Accuracy of online symptom checkers and the potential impact on service utilisation. medRxiv 2020.07.07.20147975| Accuracy of online symptom checkers and the potential impact on service utilisation.Crossref | GoogleScholarGoogle Scholar |
Chambers D, Cantrell AJ, Johnson M, Preston L, Baxter SK, Booth A, Turner J (2019) Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open 9, e027743
| Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review.Crossref | GoogleScholarGoogle Scholar | 31375610PubMed |
Cheng C, Dunn M (2015) Health literacy and the Internet: a study on the readability of Australian online health information. Australian and New Zealand Journal of Public Health 39, 309–314.
| Health literacy and the Internet: a study on the readability of Australian online health information.Crossref | GoogleScholarGoogle Scholar | 25716142PubMed |
Cooke G, Valenti L, Glasziou P, Britt H (2013) Common general practice presentations and publication frequency. Australian Family Physician 42, 65–68.
Dunn AG (2020) Will online symptom checkers improve health care in Australia? The Medical Journal of Australia
| Will online symptom checkers improve health care in Australia?Crossref | GoogleScholarGoogle Scholar | 32441062PubMed |
Elder J 2018. The robot doctor will see you now. The Sydney Morning Herald. Available at https://www.smh.com.au/lifestyle/health-and-wellness/the-robot-doctor-will-see-you-now-20180810-p4zwpy.html [Verified 6 November 2020]
Fraser H, Coiera E, Wong D (2018) Safety of patient-facing digital symptom checkers. Lancet 392, 2263–2264.
| Safety of patient-facing digital symptom checkers.Crossref | GoogleScholarGoogle Scholar | 30413281PubMed |
Gilbert S, Mehl A, Baluch A, Cawley C, Challiner J, Fraser H, Millen E, Montazeri M, Multmeier J, Pick F, Richter C, Türk E, Upadhyay S, Virani V, Vona N, Wicks P, Novorol C (2020) How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open 10, e040269
| How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs.Crossref | GoogleScholarGoogle Scholar | 33328258PubMed |
Gilbert S, Wicks P, Novorol C (2021) The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. The Medical Journal of Australia 214, 143–143.e1.
| The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia.Crossref | GoogleScholarGoogle Scholar | 33423296PubMed |
Hammond R (2019) The world in 2040. The future of healthcare, mobility, travel and the home. Future health, care and wellbeing. Allianz Partners. Available at https://www.allianz-partners.com/content/dam/onemarketing/awp/azpartnerscom/italy/futurologo/en/Allianz-Partners-The-World-in-2040-Health-Care-Wellbeing-Report1.pdf [Verified 2 May 2021]
Hill MG, Sim M, Mills B (2020) The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. The Medical Journal of Australia 212, 514–519.
| The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia.Crossref | GoogleScholarGoogle Scholar | 32391611PubMed |
Hill MG, Sim M, Mills B (2021) The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. The Medical Journal of Australia 214, 143–143.e1.
| The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia.Crossref | GoogleScholarGoogle Scholar | 33423279PubMed |
International Organization for Standardization Technical Committee 210 (ISO/TC 210) (2016) ISO 13485:2016: medical devices – quality management systems – requirements for regulatory purposes. Available at https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/05/97/59752.html [Verified 2 March 2021]
Leigh S, Ashall-Payne L (2019) The role of health-care providers in mHealth adoption. The Lancet Digital Health 1, e58–e59.
| The role of health-care providers in mHealth adoption.Crossref | GoogleScholarGoogle Scholar | 33323231PubMed |
Meyer AN, Giardina TD, Spitzmueller C, Shahid U, Scott TM, Singh H (2020) Patient Perspectives on the Usefulness of an Artificial Intelligence–Assisted Symptom Checker: Cross-Sectional Survey Study. Journal of Medical Internet Research 22, e14679
Miller S, Gilbert S, Virani V, Wicks P (2020) Patients’ Utilization and Perception of an Artificial Intelligence–Based Symptom Assessment and Advice Technology in a British Primary Care Waiting Room: Exploratory Pilot Study. JMIR Human Factors 7,
| Patients’ Utilization and Perception of an Artificial Intelligence–Based Symptom Assessment and Advice Technology in a British Primary Care Waiting Room: Exploratory Pilot Study.Crossref | GoogleScholarGoogle Scholar | 32540836PubMed |
Morse KE, Ostberg NP, Jones VG, Chan AS (2020) Use Characteristics and Triage Acuity of a Digital Symptom Checker in a Large Integrated Health System: Population-Based Descriptive Study. Journal of Medical Internet Research 22, e20549
| Use Characteristics and Triage Acuity of a Digital Symptom Checker in a Large Integrated Health System: Population-Based Descriptive Study.Crossref | GoogleScholarGoogle Scholar | 33170799PubMed |
Nateqi J, Lin S, Krobath H, Gruarin S, Lutz T, Dvorak T, Gruschina A, Ortner R (2019) Vom symptom zur diagnose – tauglichkeit von symptom-checkern. HNO 67, 334–342.
| Vom symptom zur diagnose – tauglichkeit von symptom-checkern.Crossref | GoogleScholarGoogle Scholar | 30993374PubMed |
Ramnarayan P, Cronje N, Brown R, Negus R, Coode B, Moss P, Hassan T, Hamer W, Britto J (2007) Validation of a diagnostic reminder system in emergency medicine: a multi-centre study. Emergency Medicine Journal 24, 619–624.
| Validation of a diagnostic reminder system in emergency medicine: a multi-centre study.Crossref | GoogleScholarGoogle Scholar | 17711936PubMed |
Ronicke S, Hirsch MC, Türk E, Larionov K, Tientcheu D, Wagner AD (2019) Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study. Orphanet Journal of Rare Diseases 14, 69
| Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study.Crossref | GoogleScholarGoogle Scholar | 30898118PubMed |
Rørtveit S, Meland E, Hunskaar S (2013) ) Changes of triage by GPs during the course of prehospital emergency situations in a Norwegian rural community. Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine 21, 89–89.el.
| ) Changes of triage by GPs during the course of prehospital emergency situations in a Norwegian rural community.Crossref | GoogleScholarGoogle Scholar | 24354953PubMed |
Semigran HL, Levine DM, Nundy S, Mehrotra A (2016) Comparison of physician and computer diagnostic accuracy. JAMA Internal Medicine 176, 1860–1861.
| Comparison of physician and computer diagnostic accuracy.Crossref | GoogleScholarGoogle Scholar | 27723877PubMed |
Wiegand T, Krishnamurthy R, Kuglitsch M, Lee N, Pujari S, Salathé M, Wenzel M, Xu S (2019) WHO and ITU establish benchmarking process for artificial intelligence in health. Lancet 394, 9–11.
| WHO and ITU establish benchmarking process for artificial intelligence in health.Crossref | GoogleScholarGoogle Scholar | 30935732PubMed |
Winn AN, Somai M, Fergestrom N, Crotty BH (2019) Association of Use of Online Symptom Checkers With Patients’ Plans for Seeking Care. JAMA Network Open 2, e1918561
| Association of Use of Online Symptom Checkers With Patients’ Plans for Seeking Care.Crossref | GoogleScholarGoogle Scholar | 31880791PubMed |
World Economic Forum (WEF) (2020) Chatbots RESET: a framework for governing responsible use of conversational AI in healthcare. WEF. Available at https://www.weforum.org/reports/chatbots-reset-a-framework-for-governing-responsible-use-of-conversational-ai-in-healthcare/ [Verified 2 March 2021]
