Designing evaluation plans for health promotion mHealth interventions: a case study of the Milk Man mobile app

Key words: breastfeeding, evaluation methods, evidence-based practice, information and communication technology.

In 2014 there were over 100 000 health and fitness apps available in the app stores of the two major mobile platforms (iOS and Android),¹ yet despite the proliferation of mobile health (mHealth) apps, there remains a lack of definitive evidence of the efficacy and impact mHealth apps can have in terms of reach, behaviour change and, ultimately, health outcomes. This paper discusses the need for comprehensive evaluation plans for mHealth interventions, highlights four evaluation models and tools and describes the evaluation plan of a socially connected, gamified breastfeeding app targeted at fathers – the Milk Man mobile app.

Mobile devices offer unique opportunities for health promotion professionals to design tailored interventions that make use of innovative technology to reach populations.^2–4 Physical activity interventions, for example, utilise inbuilt pedometers and accelerometers to accurately record exercise and movement,⁵ and nutrition interventions seeking to utilise food diaries can benefit from non-textual data entry, including the use of photos.⁶ Other mHealth apps incorporate strategies such as gamification, social connectivity and push notifications to reach people as they go about their daily lives.⁷

This concept of reaching people throughout their daily life is described as an Ecological Momentary Intervention (EMI).⁸ Mobile devices are a perfect fit with EMIs because peoples’ relationships with their smartphones are often intimate and constant, with almost 50% of Australians reaching for their phone within 15 min of waking.⁹ This connection to devices continues throughout the day with the average person checking in with their phone twice every hour.⁹ People rely on their devices for an increasing number of everyday tasks – banking, scheduling, connecting with friends and checking emails can all be coordinated from the one device. One UK study found smartphones were used to perform an average of 221 tasks daily, equating to 3 h and 16 min of device use every day.¹⁰ With such a strong connection, there are genuine opportunities for health behaviour–changing EMIs to reach individuals, groups and populations through their mobile device. Encouraging results have been reported for mHealth interventions designed for breastfeeding mothers,¹¹ promoting positive mental health to adolescents¹² and nutrition for adults.¹³

While apps for health appear to have great potential in reaching populations,^2–4 there remains a lack of evidence in terms of the efficacy of their use, particularly in terms of the impact on health outcomes.^14–16 The need for more evidence of the effectiveness of health apps is a common recommendation in the literature,^2,17 yet there are significant challenges in designing and implementing Randomised Control Trials (RCT) for mHealth apps. In particular, the difficulty in trialling an app over the period of time typically needed for adequately powered trials, and the potential for either changes to the technology or the technological expectations from the user over this time.¹⁸ The World Bank states mHealth services have the potential to deliver better public health outcomes at a lower cost than traditional programs, yet also states there is a lack of evidence of positive behaviour change.¹⁶ There is a need to evaluate the cost effectiveness of mHealth interventions delivered via mobile apps in comparison to other ways of reaching specific populations and motivating positive behaviour change.

An individual’s usage and engagement with apps is complex and sophisticated. Many factors can impact on the success of an mHealth intervention; to determine its success, evaluation should examine user feedback and outcome measures, as well as the robustness of the technology, the intervention principles, engagement strategies and the interaction of the user with the technology.

Evaluation is an integral part of health promotion planning and implementation¹⁹ and more knowledge and evidence is needed about what works with targeted mHealth interventions. To generate evidence to improve health promotion practice, evaluation needs to be conducted throughout the implementation of an intervention not only at the conclusion.²⁰ A multifaceted evaluation plan is required to assess the complex web of components that could impact on the success of an mHealth intervention.

As the number of mHealth initiatives has increased, researchers have developed several models and tools to guide evaluation. This paper describes four different approaches from the published literature. Two approaches described, the Collaborative Adaptive and Interactive Technologies framework and the Mobile Application Rating Scale, were incorporated into the development of the evaluation plan for the Milk Man app. More recent examples are the Trial of Intervention Principles framework and the WHO mHealth Evidence Reporting and Assessment Checklist. These did not form part of the Milk Man evaluation plan; however, the overlap in approaches is a confirmation of the growing consensus in reporting on mHealth interventions.

The Collaborative Adaptive and Interactive Technologies framework was developed by O’Grady et al. in 2009.²¹ The authors define ‘collaborative adaptive and interactive technologies’ as those technologies that facilitate collaboration between users, support adaption of content according to users’ needs and enable user interaction with the technology.²¹ This type of approach is particularly relevant for interventions that connect people, groups and populations through technology. The framework organises formative, process and impact evaluation over five key areas:

1. People – the users and stakeholders

2. Content – information or content

3. Technology – the technology used to develop and maintain the intervention

4. Computer-mediated interaction – the interactions between the user and the technology, and how the technology supports interactions between users

5. Health systems integration – how the intervention interacts with and impacts on the broader health system.

While developed with web-based interventions as the focus, the framework is comprehensive and has broad relevance to mobile apps, particularly socially connected apps that are to be trialled over a long period of time, such as Milk Man.

The MARS was developed by a multidisciplinary team as a reliable tool to assess the quality of health apps and was released in 2015.²² The scale comprises five categories, including four that measure:

1. Aesthetics – graphics, layout, visual appeal

2. Engagement – entertainment, customisation, fit to target group

3. Functionality – performance, navigation, gestural design

4. Information – quality, quantity, visual information, credibility.

The fifth category is a subjective ‘quality’ scale and asks users their opinions including if they would recommend the app to others and if they would pay for it.^22,23 The categories covered by the MARS are designed to measure a range of factors important in mobile apps and give health professionals a valid, reliable and easy-to-use tool to help them assess quality.

Mohr et al. suggest that traditional RCTs are not ideal for testing Behavioural Intervention Technologies (BIT).¹⁸ This is due to the extended time periods required for adequately powered trials that can be in direct conflict with changing technology. The Trial of Intervention Principles framework¹⁸ focuses on defining and testing the intervention principles (actual intervention aims and strategies) as opposed to a static version of the BIT that cannot be altered. By focusing on testing the intervention principles (for example, in the case of Milk Man, testing the delivery of a social support intervention for fathers via a mobile app) the model allows for some modifications to the actual BIT (improving usability and functionality, for example) as long as any changes are reported.

In 2016, the WHO mHealth Technical Evidence Review group released a checklist detailing 16 key criteria for reporting and assessing mHealth interventions.²⁴ The checklist identifies criteria needed to define the content of the intervention (what it is), the context (where it is being implemented) and the technical features (how it is being implemented). It includes criteria such as usability, cost assessment, interoperability, content and data security. The checklist was developed to help researchers develop and assess the evidence in mHealth interventions by providing a checklist for reporting the quality of the intervention (as opposed to actually evaluating the quality of the intervention).²⁴

The four approaches described are different in their intent, focus and scope yet there are distinct similarities in that they all focus on taking a multifaceted approach to evaluating mHealth interventions. Both the Trial of Intervention Principles framework and the Collaborative Adaptive Interactive Technology framework are comprehensive in scope and could be adapted for longer- or shorter-term interventions. The MARS is an easy-to-use, validated tool that can be incorporated into app development to strengthen the development process by identifying areas of weakness or strength or to rate already existing apps. The WHO mHealth example offers a concise checklist to refer to throughout the implementation of a project.

Evaluation needs and constraints will differ for each intervention. Health promotion practitioners are encouraged to consider the above approaches and design plans that incorporate components that focus on what is important and feasible for their own programs.

Milk Man is a socially connected breastfeeding app designed specifically for fathers that was developed as a strategy for use in the Parent Infant Feeding Initiative (PIFI) study.⁷ The PIFI study is a four-armed RCT that aims to increase the duration and/or exclusivity of breastfeeding.²⁵ The study involves testing two different interventions designed to increase fathers’ support for breastfeeding: a male-facilitated antenatal class and the Milk Man mobile app.

Milk Man utilises several engagement techniques including gamification (the process of embedding game-like elements in things that are not games, in the case of Milk Man, leaderboards, points and badges) and push notifications (notifications sent to users that appear on the home screen of their phone alerting them to new content).⁷ Milk Man has an extensive, searchable, evidence-based information library and a guided user-to-user conversation forum. Fathers are placed into groups depending on when their baby is due enabling age-relevant information to be pushed out twice a week through the conversation forum and fathers to be able to interact with peers with similar aged babies. The aim of the app intervention is to increase the support new and expectant fathers provide their breastfeeding partners, which we hypothesise will lead to an increase in breastfeeding duration, and in particular an increase in the duration of exclusive breastfeeding.

In developing the evaluation plan for the Milk Man app, there was a focus on planning for ongoing evaluation throughout the development of the app and through the trial. This encompassed three stages of evaluation: formative (takes place while planning interventions), process (takes place during the implementation) and impact (assesses the outcomes of the intervention) evaluation.^20,26

Milk Man is the first breastfeeding app we are aware of that targets fathers. The development of the app included formative evaluation with input from the target group and health professionals, and a user-testing phase encompassing a think-aloud walkthrough and completion of the MARS.⁷ PIFI study data are collected from participants via self-administered questionnaires at recruitment and at six and 26 weeks post-birth. Additionally data from the Milk Man app are collected via a customised analytics framework and through content analysis of the conversation forum.

The Milk Man evaluation plan is detailed in Appendix 1, and focuses broadly on evaluating the delivery of a social support intervention though a mobile app. The comprehensive plan is based on the Collaborative Adaptive Interactive Technology framework developed by O’Grady et al.²¹ and is designed to measure indicators for the formative, process and impact evaluation of the app across the five different areas – people, content, technology, computer-mediated interaction and health systems integration. As the trial is currently underway, the formative evaluation phase has been completed, with process and impact evaluation still being conducted. The five areas of evaluative focus are described below.

In the case of Milk Man ‘people’ refers to end-users (fathers) and stakeholders (health professionals). The formative evaluation phase informed the development of Milk Man and sought to identify end-users’ and stakeholders’ needs. This included an extensive literature review, focus groups and user-testing involving a think-aloud walkthrough of the app and completion of the MARS.⁷ Think-aloud walkthroughs are a common way of testing health apps where the user is observed using the app while verbalising their thought processes as they do so.²⁷ This can help to highlight issues with usability and navigation. Focus groups were conducted with end-users and stakeholders. Stakeholders were important to include at this stage as the health professionals we spoke with had direct and ongoing contact with new and expectant parents and were able to offer insight into emerging trends and how to engage fathers. The end-user groups focused on the acceptability of the proposed engagement strategies, the framing of the app and the appropriateness of the approach and content. Indicators of user perspectives were included at both the formative and process level along with individual’s intentions and motivations to use the app. Stakeholder groups focused more on the content of the library and on how best to engage the target group.

Data were gathered by the completion of the think-aloud walkthrough studies and the MARS,^22,23 and ongoing process data are being collected via questionnaires collected at six weeks postpartum as part of the RCT. The impact assessment includes examination of data gathered from the six- and 26-week study questionnaires including breastfeeding outcomes,²⁵ as well as several other constructs such as partner support²⁸ and breastfeeding self-efficacy.²⁹

The ‘content’ in the case of Milk Man refers to the static informational content, or ‘library’, contained within the app, as well as the dynamic, ever-changing, user-generated content in the conversation forum. In developing the app, formative evaluation sought to test the quality and credibility of the app content through focus groups. Understandability, functionality and usability were measured at the process stage via the think-aloud walkthrough studies, the MARS²³ and the six-week study questionnaires.

The app contains a socially connected forum for fathers to interact with each other and the impact assessment will focus on examination of the conversation content that was generated. This will begin with an assessment of the role that this forum appears to have played for users; for example, has it been of assistance for seeking advice? Has it offered emotional support? It will also examine the positioning of this forum within the app. For example, investigating if the forum was the central function that users returned to repeatedly, and used as a gateway to library information, or if they tended to find information directly in the library section, perhaps via the search function.

Technology refers to the software used to create and run the app. In this case, the formative phase included indicators such as the robustness of the system in terms of performance and speed of loading, including how it performed with multiple users. It also involved examining the two different operating systems employed by the study (iOS and Android) and examining the similarities and differences. Privacy of users was an important consideration including how data were stored and coded to maintain research integrity and confidentiality. This included the development of a robust set of management protocols for the app. Process evaluation will involve an examination of the analytics framework embedded within the app to answer questions such as, how people are using the app, which sections are the most popular and at what age of their child, or developmental milestones, are fathers most often seeking information and support. The impact assessment will involve looking at the dynamic evolution of the app (how it responds to new technologies or social trends) over time, which is particularly important in this study as the app is being trialled over a two-year period. By examining how the app responds to operating system updates and other technological changes the robustness of the technology over time can be monitored, and the impact this may have on implementation can be considered.

Computer-mediated technology describes the interaction of users with the technological interface. Formative evaluation primarily included the think-aloud walkthrough studies that measure the usability of the app, how the information is organised and how intuitive this is for the user. Further information is sought from users about their perspectives on the usability, findability and sociability of the app through the app-specific questions included in the six- and 26-week study questionnaires. The impact assessment will use app analytics and content analysis of the forum to explore the ‘community development’, how the app has supported the interaction of the ‘community’ or the users on the study over time, and how users have engaged with each other and with the technology.

Health systems integration represents the larger system in which the intervention may be implemented.²¹ In this case the formative evaluation involved the development of a comprehensive evaluation framework, and consideration of ethical issues. Process evaluation will assess the impact of the app on participants’ usage of other health services. This will include examining visits to external websites from within the app, and data collected via the study questionnaires that asks participants about the health professionals they have consulted in relation to breastfeeding. The impact assessment of the Milk Man app will include consideration of its sustainability and how it can be embedded into existing health services. Finally, the PIFI study includes an economic analysis of the cost effectiveness of the two different interventions being trialled.²⁵

As mobile technologies continue to become integrated into the daily lives of individuals and populations, health promotion practitioners will increasingly require the ability to plan effective mHealth interventions. Challenges remain in the development of mHealth interventions, including privacy and data security considerations, the need for more evidence about their efficacy and demonstrated pathways for sustainability beyond funded projects. Incorporating comprehensive evaluation into program planning will help ensure the evidence base continues to grow.

The Milk Man case study describes a comprehensive evaluation approach that will provide evidence from an adequately powered RCT to inform future mHealth interventions. While this detailed approach will not be practical for every intervention, if practitioners and researchers continue to think broadly about how they can incorporate ongoing evaluation across a range of factors, the sector will be best placed to grow the evidence.

The technology sector moves and iterates quickly. As more health promotion practitioners and researchers seek to incorporate mHealth into interventions, there is a need to ensure the skills and knowledge of the workforce remain up-to-date with new technologies. The tools and case study described here can help guide health promotion practitioners working in mHealth to develop effective and thorough evaluation plans. Developing comprehensive plans, such as the one described in this paper for the Milk Man app, will help ensure that the knowledge is transferable and relevant across disciplines and move health promotion research towards a greater understanding of how mHealth can be best used.