Designing evaluation plans for health promotion mHealth interventions: a case study of the Milk Man mobile app
Becky K. White A D , Sharyn K. Burns A B , Roslyn C. Giglia A C and Jane A. Scott A B
+ Author Affiliations
- Author Affiliations
A School of Public Health, Curtin University, GPO Box U1987, Bentley, Perth, WA 6845, Australia.
B Collaboration for Evidence, Research and Impact in Public Health (CERIPH), Faculty of Health Sciences, Curtin University, PO Box U1987, Bentley, WA 6845, Australia.
C Telethon Kids Institute, University of Western Australia, 100 Roberts Road, Subiaco, WA 6008, Australia.
D Corresponding author. Email: becky.white@curtin.edu.au
Health Promotion Journal of Australia 27(3) 198-203 https://doi.org/10.1071/HE16041
Submitted: 5 May 2016 Accepted: 7 September 2016 Published: 26 October 2016
Journal Compilation © Australian Health Promotion Association 2016
Abstract
Evaluating complex health promotion interventions that use mobile apps requires comprehensive and adaptive evaluation plans. As mobile usage becomes increasingly sophisticated and personalised, broad evaluation plans are important in determining the impact and efficacy of a mobile health (mHealth) app. Evaluation should consider user feedback and outcome measures, as well as examine elements such as the robustness of the technology, the intervention principles and engagement strategies, and the interaction of the user with the technology. This paper introduces four mHealth evaluation models and tools and describes the evaluation plan that has been developed for Milk Man, a breastfeeding app targeting new and expectant fathers. Milk Man is a socially connected, gamified app that is being tested in a large Randomised Control Trial (RCT). While there is a need for mobile apps to be evaluated in adequately powered RCTs, trialling mobile apps over a long period of time presents challenges. Incorporating robust evaluation design will help ensure that technological performance, app intervention principles, as well as health and behavioural outcomes are measured. The detail and scope of the Milk Man app evaluation plan will ensure the findings add to the evidence base and have broad relevance to health promotion practitioners.
So what?Evidence about the efficacy of mHealth interventions is an emerging area and appropriate evaluation skills are needed. This paper illustrates an evaluation planning approach for mHealth interventions that could be adapted for use by health promotion practitioners and researchers.
Key words: breastfeeding, evaluation methods, evidence-based practice, information and communication technology.
References
[1] Research2guidance. mHealth app developer economics 2014: the state of the art of mHealth app publishing. 2014. Available from: http://www.research2guidance.com/r2g/research2guidance-mHealth-App-Developer-Economics-2014.pdf [Verified 10 April 2015].[2] Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al (2013) The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 10, e1001362
| The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review.Crossref | GoogleScholarGoogle Scholar | 23349621PubMed |
[3] Kratzke C, Cox C (2012) Smartphone technology and apps: rapidly changing health promotion. Int Electron J Health Educ 15, 72–82.
[4] Becker S, Miron-Shatz T, Schumacher N, Krocza J, Diamantidis C, Albrecht U (2014) mHealth 2.0: experiences, possibilities, and perspectives. JMIR Mhealth Uhealth 2, e24
| mHealth 2.0: experiences, possibilities, and perspectives.Crossref | GoogleScholarGoogle Scholar | 25099752PubMed |
[5] Fanning J, Mullen SP, McAuley E (2012) Increasing physical activity with mobile devices: a meta-analysis. J Med Internet Res 14, e161
| Increasing physical activity with mobile devices: a meta-analysis.Crossref | GoogleScholarGoogle Scholar | 23171838PubMed |
[6] Boushey CJ, Harray AJ, Kerr DA, Schap TE, Paterson S, Aflague T, et al (2015) How willing are adolescents to record their dietary intake? The mobile food record. JMIR Mhealth Uhealth 3, e47
| How willing are adolescents to record their dietary intake? The mobile food record.Crossref | GoogleScholarGoogle Scholar | 26024996PubMed |
[7] White BK, Martin A, White JA, Burns SK, Maycock BR, Giglia RC, et al (2016) Theory-based design and development of a socially connected, gamified mobile app for men about breastfeeding (Milk Man). JMIR Mhealth Uhealth 4, e81
| Theory-based design and development of a socially connected, gamified mobile app for men about breastfeeding (Milk Man).Crossref | GoogleScholarGoogle Scholar | 27349756PubMed |
[8] Heron KE, Smyth JM (2010) Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. Br J Health Psychol 15, 1–39.
| Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments.Crossref | GoogleScholarGoogle Scholar | 19646331PubMed |
[9] Deloitte. Mobile consumer survey 2014. The Australian cut: revolution and evolution. 2014. Available from: http://landing.deloitte.com.au/rs/deloitteaus/images/Deloitte_Mobile_Consumer_Survey_2014.pdf [Verified 22 April 2016].
[10] Tecmark. Tecmark survey finds average user picks up their smartphone 221 times a day. 2014. Available from: http://www.tecmark.co.uk/smartphone-usage-data-uk-2014/ [Verified 9 April 2016].
[11] White B, White J, Giglia R, Tawia S (2016) Feed safe: a multidisciplinary partnership approach results in a successful mobile application for breastfeeding mothers. Health Promot J Austr 27, 111–7.
| Feed safe: a multidisciplinary partnership approach results in a successful mobile application for breastfeeding mothers.Crossref | GoogleScholarGoogle Scholar |
[12] Kenny R, Dooley B, Fitzgerald A (2015) Feasibility of “CopeSmart”: a telemental health app for adolescents. JMIR mental health 2, e22
| Feasibility of “CopeSmart”: a telemental health app for adolescents.Crossref | GoogleScholarGoogle Scholar | 26552425PubMed |
[13] Gilliland J, Sadler R, Clark A, O’Connor C, Milczarek M, Doherty S (2015) Using a smartphone application to promote healthy dietary behaviours and local food consumption. Biomed Res Int 2015, 841368
| Using a smartphone application to promote healthy dietary behaviours and local food consumption.Crossref | GoogleScholarGoogle Scholar | 26380298PubMed |
[14] Agarwal S, Perry HB, Long LA, Labrique AB (2015) Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Trop Med Int Health 20, 1003–14.
| Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review.Crossref | GoogleScholarGoogle Scholar | 25881735PubMed |
[15] Agarwal S, Labrique A (2014) Newborn health on the line: the potential mHealth applications. JAMA 312, 229–30.
| Newborn health on the line: the potential mHealth applications.Crossref | GoogleScholarGoogle Scholar | 1:CAS:528:DC%2BC2cXhs1egtbbI&md5=d2b852296a8a3ef8455ae4ad3f61fb62CAS | 24953141PubMed |
[16] Qiang CZYM, Hausman V, Altman DG. Mobile applications for the health sector. 2011. Available from: http://siteresources.worldbank.org/INFORMATIONANDCOMMUNICATIONANDTECHNOLOGIES/Resources/mHealth_report.pdf [Verified 22 April 2016].
[17] Baskerville NB, Struik LL, Hammond D, Guindon GE, Norman CD, Whittaker R, et al (2015) Effect of a mobile phone intervention on quitting smoking in a young adult population of smokers: randomized controlled trial study protocol. JMIR Res Protoc 4, e10
| Effect of a mobile phone intervention on quitting smoking in a young adult population of smokers: randomized controlled trial study protocol.Crossref | GoogleScholarGoogle Scholar | 25599695PubMed |
[18] Mohr DC, Schueller SM, Riley WT, Brown CH, Cuijpers P, Duan N, et al (2015) Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res 17, e166
| Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies.Crossref | GoogleScholarGoogle Scholar | 26155878PubMed |
[19] O’Connor-Fleming ML, Parker E, Higgins H, Gould T (2006) A framework for evaluating health promotion programs. Health Promot J Austr 17, 61–6.
| A framework for evaluating health promotion programs.Crossref | GoogleScholarGoogle Scholar | 16619938PubMed |
[20] Lobo R, Burns SK, Petrich M (2014) Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health 14, 1315
| Supporting health promotion practitioners to undertake evaluation for program development.Crossref | GoogleScholarGoogle Scholar | 25532523PubMed |
[21] O’Grady L, Witteman H, Bender J, Urowitz S, Wiljer D, Jadad A (2009) Measuring the impact of a moving target: towards a dynamic framework for evaluating collaborative adaptive interactive technologies. J Med Internet Res 11, e20
| Measuring the impact of a moving target: towards a dynamic framework for evaluating collaborative adaptive interactive technologies.Crossref | GoogleScholarGoogle Scholar | 19632973PubMed |
[22] Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M (2015) Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 3, e27
| Mobile app rating scale: a new tool for assessing the quality of health mobile apps.Crossref | GoogleScholarGoogle Scholar | 25760773PubMed |
[23] Hides L, Kavanagh D, Stoyanov S, Zelenko O, Tjondroegoro D, Mani M. Mobile Application Rating Scale (MARS): a new tool for assessing the quality of health mobile applications. Melbourne: Young and Well Cooperative Research Centre; 2014.
[24] Agarwal S, Lefevre AE, Lee J, L’Engle K, Mehl G, Sinha C, et al (2016) Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 352, i1174
| Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist.Crossref | GoogleScholarGoogle Scholar | 26988021PubMed |
[25] Maycock BR, Scott JA, Hauck YL, Burns SK, Robinson S, Giglia R, et al (2015) A study to prolong breastfeeding duration: design and rationale of the Parent Infant Feeding Initiative (PIFI) randomised controlled trial. BMC Pregnancy Childbirth 15, 159
| A study to prolong breastfeeding duration: design and rationale of the Parent Infant Feeding Initiative (PIFI) randomised controlled trial.Crossref | GoogleScholarGoogle Scholar | 26231519PubMed |
[26] Howat P, Brown G, Burns S, McManus A. Project Planning, using the Precede Proceed Model. In Jirowong S, LIamputtong P, editors. Population health, communities and health promotion. Sydney: Oxford University Press; 2009.
[27] Jaspers MWM (2009) A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform 78, 340–53.
| A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence.Crossref | GoogleScholarGoogle Scholar |
[28] Dennis C-L., Ross L (2006) Women’s perceptions of partner support and conflict in the development of postpartum depressive symptoms. J Adv Nurs 56, 588–99.
| Women’s perceptions of partner support and conflict in the development of postpartum depressive symptoms.Crossref | GoogleScholarGoogle Scholar | 17118038PubMed |
[29] Dennis CL (2003) The breastfeeding self-efficacy scale: psychometric assessment of the short form. J Obstet Gynecol Neonatal Nurs 32, 734–44.
| The breastfeeding self-efficacy scale: psychometric assessment of the short form.Crossref | GoogleScholarGoogle Scholar | 14649593PubMed |
