Systematic review of the types of methods and approaches used to assess the effectiveness of healthcare information websites
Jennifer Tieman A B and Sandra L. Bradley A
+ Author Affiliations
- Author Affiliations
A Palliative and Supportive Services, School of Medicine, Flinders University, Health Sciences Building, Repatriation General Hospital Campus, Daw Park, SA 5041, Australia.
B Corresponding author. Email: Jennifer.Tieman@flinders.edu.au
Australian Journal of Primary Health 19(4) 319-324 https://doi.org/10.1071/PY13030
Submitted: 28 February 2013 Accepted: 6 August 2013 Published: 5 September 2013
Abstract
The aim of this systematic review was to identify types of approaches and methods used to evaluate the effectiveness of healthcare information websites. Simple usage data may not be sufficient to assess whether desired healthcare outcomes were achieved or to determine the relative effectiveness of different web resources on the same health topic. To establish the state of the knowledge base on assessment methods used to determine the effectiveness of healthcare websites, a structured search of the literature was conducted in Ovid Medline, resulting in the retrieval of 1611 articles, of which 240 met the inclusion criteria for the present review. The present review found that diverse evaluation methods were used to measure the effectiveness of healthcare websites. These evaluation methods were used during development, before release and after release. Economic assessment was rare and most evaluations looked at content issues, such as readability scores. Several studies did try to assess the usefulness of websites, but few studies looked at behaviour change or knowledge transfer following engagement with the designated health website. To assess the effectiveness of the knowledge transfer of healthcare information through the online environment, multiple methods may need to be used to evaluate healthcare websites and may need to be undertaken at all stages of the website development process.
References
Australian Bureau of Statistics (2012) ‘Internet activity, Australia.’ (Australian Bureau of Statistics: Canberra) Available at http://www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/8153.0Main+Features1Jun%202012?OpenDocument [Verified 21 February 2013]Australian Government (2010) ‘Building a 21st century primary health care system: Australia’s first national primary health care strategy.’ (Department of Health and Ageing) Available at http://www.yourhealth.gov.au/internet/yourhealth/publishing.nsf/Content/report-primaryhealth/$File/NPHCS-Foreword-intro.pdf [Verified 21 February 2013]
Australian Government (2012) ‘The Australian Government web guide.’ (Department of Finance and Deregulation) Available at http://webguide.gov.au/ [Verified 25 February 2013]
Australian Government Information Office (2013) ‘Better practice checklists & guidance.’ Available at http://agict.gov.au/policy-guides-procurement/better-practice-checklists-guidance/ [Verified 27 July 2013]
Boucher J (2010) Technology and patient–provider interactions: improving quality of care, but is it improving communication and collaboration? Diabetes Spectrum 23, 142–144.
| Technology and patient–provider interactions: improving quality of care, but is it improving communication and collaboration?Crossref | GoogleScholarGoogle Scholar |
Calero C, Ruiz J, Piattini M (2005) Classifying web metrics using the web quality model. Online Information Review 29, 227–248.
| Classifying web metrics using the web quality model.Crossref | GoogleScholarGoogle Scholar |
Catwell L, Sheikh A (2009) Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Medicine 6, e1000126
| Evaluating eHealth interventions: the need for continuous systemic evaluation.Crossref | GoogleScholarGoogle Scholar | 19688038PubMed |
Choi NG, Dinitto DM (2013) The digital divide among low-income homebound older adults: Internet use patterns, eHealth literacy, and attitudes toward computer/Internet use. Journal of Medical Internet Research 15, e93
| The digital divide among low-income homebound older adults: Internet use patterns, eHealth literacy, and attitudes toward computer/Internet use.Crossref | GoogleScholarGoogle Scholar | 23639979PubMed |
Cunliffe D (2000) Developing usable websites: a review and a model. Internet Research 10, 295–308.
| Developing usable websites: a review and a model.Crossref | GoogleScholarGoogle Scholar |
Ekeland AG, Bowes A, Flottorp S (2012) Methodologies for assessing telemedicine: a systematic review of reviews. International Journal of Medical Informatics 81, 1–11.
| Methodologies for assessing telemedicine: a systematic review of reviews.Crossref | GoogleScholarGoogle Scholar | 22104370PubMed |
Elling S, Lentz L, de Jong M (2007) Website evaluation questionnaire. Development of a research-based tool for evaluating informational websites. In ‘Proceedings of the 6th International Conference EGOV 2007, LNCS 4656’. (Eds MA Wimmer, HJ Scholl, A Grönlund) pp. 293–304.
Eysenbach G (2011) CONSORT E-HEALTH: improving and standardizing evaluation reports on web-based and mobile health interventions. Journal of Medical Internet Research 13, e126
| CONSORT E-HEALTH: improving and standardizing evaluation reports on web-based and mobile health interventions.Crossref | GoogleScholarGoogle Scholar | 22209829PubMed |
Eysenbach G, Kohler C (2003) What is the prevalence of health-related searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the Internet. AMIA Annual Symposium Proceedings 2003, 225–229.
Fox S, Duggan M (2013) ‘Health online 2013.’ Pew Internet & American Life Project. Available at http://pewinternet.org/Reports/2013/Health-online.aspx [Verified 21 February 2013]
Glynn RW, Kelly JC, Coffey N, Sweeney KJ, Kerin MJ (2011) The effect of breast cancer awareness month on internet search activity: a comparison with awareness campaigns for lung and prostate cancer. BMC Cancer 11, 442
| The effect of breast cancer awareness month on internet search activity: a comparison with awareness campaigns for lung and prostate cancer.Crossref | GoogleScholarGoogle Scholar | 21993136PubMed |
Greenhalgh T, Russell J (2010) Why do evaluations of ehealth programs fail? An alternative set of guiding principles. PLoS Medicine 7, e1000360
| Why do evaluations of ehealth programs fail? An alternative set of guiding principles.Crossref | GoogleScholarGoogle Scholar | 21072245PubMed |
Internet World Stats (2013) ‘World Internet usage and population statistics june 30, 2012.’ Available at http://www.internetworldstats.com/stats.htm [Verified 21 February 2013]
Kruse RL, Koopman RJ, Wakefield BJ, Wakefield DS, Keplinger LE, Canfield SM, Mehr DR (2012) Internet use by primary care patients: where is the digital divide? Family Medicine 44, 342–347.
McMullan M (2006) Patients using the Internet to obtain health information: how this affects the patient–health professional relationship. Patient Education and Counseling 63, 24–28.
| Patients using the Internet to obtain health information: how this affects the patient–health professional relationship.Crossref | GoogleScholarGoogle Scholar | 16406474PubMed |
Pagliari C (2007) Design and evaluation in eHealth: challenges and implications for an interdisciplinary field. Journal of Medical Internet Research 9, e15
| Design and evaluation in eHealth: challenges and implications for an interdisciplinary field.Crossref | GoogleScholarGoogle Scholar | 17537718PubMed |
Patton MQ (2008) ‘Utilization-focused evaluation’, 4th edn. (Sage: Thousand Oaks, CA)
Pawson R, Wong G, Owen L (2011) Known knowns, known unknowns, unknown unknowns: the predicament of evidence-based policy. The American Journal of Evaluation 32, 518–546.
| Known knowns, known unknowns, unknown unknowns: the predicament of evidence-based policy.Crossref | GoogleScholarGoogle Scholar |
Reinfeld-Kirkman N, Kalucy E, Roeger L (2010) The relationship between self-reported health status and the increasing likelihood of South Australians seeking Internet health information. Australian and New Zealand Journal of Public Health 34, 422–426.
| The relationship between self-reported health status and the increasing likelihood of South Australians seeking Internet health information.Crossref | GoogleScholarGoogle Scholar | 20649784PubMed |
Rubenstein EL (2012) ‘Things my doctor never told me’: Bridging information gaps in an online community. American Society for Information Science and Technology. Available at https://www.asis.org/asist2012/proceedings/Submissions/126.pdf [Verified 27 July 2013]
Sillence E, Briggs P, Harris PR, Fishwick L (2007) How do patients evaluate and make use of online health information? Social Science & Medicine 64, 1853–1862.
| How do patients evaluate and make use of online health information?Crossref | GoogleScholarGoogle Scholar |
SPSS Inc. (2010) ‘IBM SPSS statistics (Version 19.0.0.1).’ (IBM Corporation: Somers, NY)
Tankeleviviene L, Damasevicius R (2009) Characteristics of domain ontologies for web based learning and their application for quality evaluation. Informatics in Education 8, 131–152.
Tieman J, Martin P (2009) Metrics, measures and meanings: evaluating the CareSearch website. In ‘Positioning the profession: the Tenth International Congress on Medical Librarianship’, Brisbane, Australia, 31 August–4 September 2009. Available at http://espace.library.uq.edu.au/eserv/UQ:179705/n2_2_Wed_Tieman_77.pdf [Verified 27 July 2013]
US Department of Health and Human Services (2006) ‘The research-based web design and usability guidelines, enlarged/expanded edition.’ (US Government Printing Office: Washington DC)
Yusof MM, Papazafeiropouloub A, Pualb RJ, Stergioulasb LK (2008) Investigating evaluation frameworks for health information systems. International Journal of Medical Informatics 77, 377–385.
| Investigating evaluation frameworks for health information systems.Crossref | GoogleScholarGoogle Scholar | 17904898PubMed |
