Health literacy measurement: a comparison of four widely used health literacy instruments (TOFHLA, NVS, HLS-EU and HLQ) and implications for practice

Introduction

Over the past three decades, the conceptualisation of health literacy has evolved from a focus on an individuals’ attributes and skills toward an interactive process, influenced and mediated by supportive relationships at home and with health providers and the health system at large. Actions to develop the health literacy of individuals have subsequently broadened from initiatives that support reading, comprehension, and numeracy skill development to health system improvements that support people to access, understand, appraise, remember, and use health information and services to promote and manage their health (Nutbeam 2000, 2021).

There are now several widely used instruments to measure health literacy, but each instrument has been developed from different conceptions of health literacy (Jordan et al. 2011). These conceptual differences relate to the measurement purpose of each instrument (Hawkins et al. 2021). Subsequently, the differences between what each instrument measures can present challenges for researchers and clinicians when deciding which instrument will generate data that are appropriate, meaningful, and useful for their research or clinical purpose.

Instruments that measure health literacy fall largely into one of two categories: first generation performance-based measures that assess functional health literacy, and more recent self-report measures that aim to capture health literacy as a multidimensional construct (i.e. include items beyond reading and writing ability) but include measures of functional health literacy in self-report format. Performance-based measures of health literacy determine how a person objectively performs on a test (in this case, of numeracy and/or reading comprehension) while self-report instruments ask an individual to respond about their own perception of their abilities, behaviours, attitudes, and experiences. Two of the most widely used performance-based measures of health literacy are the Test of Functional Health Literacy in Adults (TOFHLA) (Parker et al. 1995) and the Newest Vital Sign (NVS) (Weiss et al. 2005), while two of the most widely used multidimensional self-report measures are the European Health Literacy Survey (HLS–EU-47) (Sørensen et al. 2013) and the Health Literacy Questionnaire (HLQ) (Osborne et al. 2013).

The newer multidimensional self-report instruments measure a broader concept of health literacy than earlier instruments, but the data from these different types of instruments are often interpreted in similar ways and used for similar decision-making purposes (Jordan et al. 2011; Haun et al. 2012). Data interpretations are often directed at determining a ‘level’ of health literacy (e.g. high or low, adequate or inadequate), with a focus on using data to identify what are seen as health literacy deficits in individuals or groups and populations (Nutbeam 2009; Nutbeam and Muscat 2020), and inform (usually clinical) practice (Al Sayah et al. 2013; Matsuoka et al. 2016). However, there is a growing global momentum to focus on the health literacy assets (i.e. strengths) of individuals and communities and to address community-identified health literacy needs (Nutbeam 2009; Dias et al. 2021; Hawkins et al. 2021; Osborne et al. 2022a). This shift in the use of health literacy data necessitates an understanding of the different constructs captured by the different instruments to ensure that data-derived inferences are appropriate, meaningful, and useful for the intended purpose. Similarly, it must be determined if validity evidence can verify the use of a health literacy instrument for a particular measurement purpose and context. For example, to understand the full range of health literacy competencies in a given population, there must be evidence that an instrument can differentiate between people with different competencies. If the purpose of measurement is to measure change in an individual’s health literacy over time, then there must be evidence that an instrument demonstrates adequate breadth, reliability, and sensitivity to be able to measure this change.

To date, there are no studies that have compared the performance of these health literacy instruments in a head-to-head study. The aim of this study was to compare the TOFHLA, NVS, HLS-EU-47, and HLQ using internal consistency and floor and ceiling effects, and to explore the correlation between measures.

Materials and methods

This is a within-subject study comparing related health literacy measures within the same patient. Patients attending Northern Health, a large metropolitan hospital in Melbourne, Australia, for day hospital procedures (including infusions, transfusions, venesection, and trial of void) were invited to participate (including people from migrant and refugee backgrounds). The hospital is located in an area of high socioeconomic disadvantage with a higher number of migrants and refugees, and higher rates of unemployment, than State averages (Jessup et al. 2022; Jessup et al. 2023).

All consecutive eligible patients were approached on 2 days per week for the study duration. Instruments were administered at a time that was convenient to their care and the needs of the clinical staff. Exclusion criteria were <18 years of age, cognitive impairment, and those unable to converse in English. Participants were first approached by a nurse on the ward, who would advise the lead author if the patient was capable, comfortable, and willing to participate. The lead author then approached the patient, explained the study, and gained written informed consent. Recruitment was consecutive until minimum sample size was reached. The instruments were administered in random order to prevent administration order bias. Participation was voluntary and participants were informed that they could choose to end their participation if they were fatigued or distressed. In cases where participants were unable to self-administer an instrument but they were still interested in participating, the lead author verbally administered the HLS-EU-Q47 and HLQ only (as these instruments could be administered in interview form).

Instruments

In 2000, Nutbeam proposed a three-tiered model to describe individual health literacy skills:

• Functional health literacy: the basic skills required to read health information;

• Interactive/communicative health literacy: the advanced cognitive, literacy, and social skills that allow an individual to interpret, derive meaning of, and apply health information;

• Critical health literacy: the advanced skills, which, together with social skills, can be used to analyse information related to health and allow individuals to exert greater control over their circumstances.

This model suggests that health literacy skills are on a continuum. Development of an individual’s health literacy may need to also consider additional concepts including (from the practice-based World Health Organization (WHO) model for health literacy development (World Health Organization 2022)):

• Health literacy responsiveness: the extent to which health workers, services, systems, organisations, and policy makers recognise and accommodate diverse health literacy strengths and needs.

• Community health literacy: centred around family, peer, and other community conversations, and reliant on developing community assets related to health literacy.

Table 1 displays an overview of the four health literacy instruments chosen for this study, including instrument purpose (as described by developers), mode of delivery, and time to complete. In addition, each has been mapped to both the Nutbeam (2000) and WHO (Osborne et al. 2022b) models of health literacy to provide an overview of the type of ‘health literacy’ information that is gathered by each instrument. Each individual instrument is outlined in greater detail below.

Table 1.Newest Vital Sign (NVS), Test of Functional Health Literacy in Adults TOFHLA), European Health Literacy survey (HLS-EU-Q47), and the Health Literacy Questionnaire (HLQ): instrument purpose, skills measured, and contexts for health literacy development.

	Nutbeam model of health literacy skills (2000)			WHO model for health literacy development (2021)			Format
	Functional	Interactive	Critical	Individual	Responsiveness	Community
TOFHLA measurement purpose: To assess functional health literacy (Parker et al. 1995) Administration mode: Timed test Administration time: 22 min Scoring: Summary score out of 100 combining reading comprehension and numeracy scales. Adequate health literacy if the TOFHLA score is 75–100, marginal health literacy if it is 60–74, and inadequate health literacy if the score is 0–59.
Reading comprehension	✓			✓			Correct answers according to developer instructions
Numeracy	✓			✓			Cloze procedure
NVS measurement purpose: To rapidly screen for limited literacy (Weiss et al. 2005) Administration mode: Timed test Administration time: 3–6 min Scoring: Summary score out of 6, with 0–1 high likelihood of limited literacy, 2–3 possibility of limited health literacy, and 4–6 adequate health literacy.
Literacy (reading, interpretation, numeracy)	✓			✓			Correct answers using score card
HLS-EU-Q47 measurement purpose: To measure the health literacy (3 domains) of populations (Sørensen et al 2013) Administration mode: Interview or self-administered Administration time: 20–30 min Scoring: Summary score, with 0–25 indicating inadequate, >25–33 indicating problematic health literacy, >33–42 indicating sufficient health literacy and >42–50) indicating excellent health literacy.
1. Healthcare	✓			✓			Five-point response options: 1 = very difficult 2 = difficult 3 = easy 4 = very easy 5 = (don’t know – interviewer only)
2. Disease prevention		✓		✓
3. Health promotion			✓	✓		✓
HLQ measurement purpose: To measure profiles of health literacy strengths and needs (across nine domains) for population health surveys and to inform the development and evaluation of health literacy interventions (Osborne et al. 2013) Administration mode: Interview or self-administered Administration time: 7–30 min Scoring: No summary score or classification of health literacy level. HLQ provides nine individual scores based on an average of the items within each of the nine scales.
1. Feeling understood and supported by healthcare providers		✓		✓	✓		Four-point response options: 1 = strongly disagree 2 = disagree 3 = agree 4 = strongly agree
2. Having sufficient information to manage my health	✓			✓
3. Actively managing my health		✓	✓	✓
4. Social support for health		✓	✓	✓		✓
5. Appraisal of health information			✓	✓	✓
6. Ability to actively engage with healthcare providers		✓		✓	✓		Five-point response options: 1 = cannot do or always difficult 2 = usually difficult 3 = sometimes difficult 4 = usually easy 5 = always easy
7. Navigating the healthcare system				✓	✓
8. Ability to find good health information	✓	✓		✓	✓
9. Understand health information well enough to know what to do	✓			✓

The Test of Functional Health Literacy in Adults (TOFHLA)

The TOFHLA was developed to provide physicians with an objective, performance-based measure of functional health literacy with the goal of identifying patients’ ability to read health-related materials (Parker et al. 1995). It consists of two scales: a 50-item literacy test assessing reading comprehension (TOFHLA Reading comprehension), and a 17-item numeracy test (TOFHLA Numeracy). The TOFHLA Reading comprehension scale uses a modified Cloze procedure: every 5th to 7th word in a passage is omitted and four multiple-choice options are provided, one of which is correct and three of which are similar but grammatically or contextually incorrect. An example of one of the TOHLA reading comprehension scale questions is provided in Fig. 1.

Fig. 1.

TOFHLA reading comprehension scale example question.

The 17-item TOFHLA Numeracy scale uses hospital forms and prescription labels to test comprehension. The Australian version of the TOFHLA was used in this study (Barber et al. 2009). The TOFHLA Numeracy scale is weighted to create a score from 0 to 50. This is then added to the TOFHLA Reading comprehension scale score (also scored 0 to 50) yielding a score from 0 to 100. An example of a question in the TOFHLA Numeracy scale is provided in Fig. 2.

Fig. 2.

TOFHLA Numeracy scale example question.

The Newest Vital Sign (NVS)

The NVS was developed by a panel of experts in health literacy (Weiss et al. 2005). The test was designed to provide a quick screen for limited functional health literacy, with the goal of understanding which patients have limited literacy in the primary healthcare setting. The NVS consists of a nutrition label from an ice cream container accompanied by six questions. There are four numeracy questions that ask the participant to either interpret the label or calculate equations using the information on the label. Two further questions can be administered if one of the first four questions is answered incorrectly. These two questions assess the reader’s ability to comprehend the product contents written on the label. The label and an example of one of the questions are outlined in Fig. 3.

Fig. 3.

Newest vital sign example question.

The European Health Literacy Survey (HLS-EU-Q47)

The HLS-EU-Q47 was developed by the European Health Literacy Consortium following a review of the literature and development of a theoretical framework (Sørensen et al. 2013). The goal of the instrument is to comprehensively measure health literacy in populations. Items address aspects of functional, interactive, and critical health literacy in one scale across three domains. The instrument can be self-completed or administered by interview and has 47 items across the domains that cover elements of accessing, understanding, appraising, and applying information. An overall score is calculated with cut-off thresholds for inadequate, problematic, sufficient, or excellent health literacy. Data about the health literacy of individuals are collated and interpreted as the ability of populations to access, understand, appraise, and apply health information across the three domains: Healthcare (functional health literacy); Disease prevention (some items include aspects of interactive health literacy); and Health promotion (some items include aspects of critical health literacy). Items in the Healthcare and Disease prevention scales relate to settings in which individual health literacy is measured. Some items in the Health promotion scale relate to a community health literacy setting in terms of what individuals know about health education and promotion in their community. An overall health literacy score and separate scale scores are calculated for each participant. Scores are standardised on a scale ranging from 0 to 50. An example of a question from the Healthcare scale is provided in Fig. 4.

Fig. 4.

Example question from HLS-EU-Q47 domain 3. Healthcare.

The Health Literacy Questionnaire (HLQ)

The HLQ was developed using a grounded validity-driven approach to capture the multidimensional construct of health literacy from both patient and clinician perspectives (Osborne et al. 2013), with the goal of creating a measure that reflects the multidimensional nature of health literacy. The grounded approach consisted of interviews and concept mapping workshops with community members, health practitioners, hospital managers, and policy makers. The nine HLQ scales are independent questionnaires and result in nine scale scores – a single total score is not generated (Elsworth et al. 2016). There are 44 questions across the following nine scales: 1. Feeling understood and supported by health care providers; 2. Having sufficient information to manage my health; 3. Actively managing my health; 4. Social support for health; 5. Appraisal of health information; 6. Ability to actively engage with healthcare providers; 7. Navigating the healthcare system; 8. Ability to find good health information; and 9. Understand health information well enough to know what to do. Item scores within each scale are summed and divided by the number of items in the scale with all items having equal weighting. The HLQ scale scores do not have cut-off points: a lower scale score indicates potential need in relation to the scale construct while a higher score indicates a potential strength (Beauchamp et al. 2015). Data are used to identify profiles of health literacy strengths, needs, and preferences of individuals, communities, and populations. While items in all nine HLQ scales relate to individual health literacy, some scales have one or more items that relate to health literacy responsiveness and community health literacy. Fig. 5 provides an example of a question from scale 9. Understand health information well enough to know what to do.

Fig. 5.

Example question from the HLQ scale 9. Understand health information well enough to know what to do.

Results

Fifty-nine of 100 consecutive patients approached consented to participate. Three participants were excluded from the final analysis due to missing data. Mean age of participants was 57 years (s.d. 15, range 32 to 86), 33 were female (58%) (Table 2). Non-participants were slightly younger with a mean age of 57 years (s.d. 18, (range 18 to 91) and slightly more (68%) were female. No further information was available for non-participants.

Forty-nine participants completed three or more instruments, with 43 participants completing all four. Seven participants completed two instruments. Respondent fatigue was the primary reason for non-completion, with time to complete all four instruments ranging from 41 minutes to 2 hours. In addition, among those completing only two instruments, five were unable to complete the NVS and TOFHLA due to visual impairments or illiteracy. Four participants partially completed three or more instruments and their data were excluded.

All scales demonstrated satisfactory to high CR (CR range 0.76–0.95) and internal consistency reliability (α > 0.77) (Table 3). The HLS-EU-Q47 Health promotion scale had the highest CR (0.95 [CI 0.91 to 0.96], 16 items). The TOFHLA Numeracy scale had the lowest CR (0.76 [CI 0.59 to 0.83], 17 items). Cronbach’s alpha and CR were similar across the scales.

Table 4 provides an overview of the number of participants who completed each scale, the mean and median scores for the scale, skewness and kurtosis, and floor and ceiling effects for all scales within each instrument (histograms are also available in the Supplementary material). More participants were able to complete the self-report instruments, HLS-EU-Q47 (n = 52) and the HLQ (n = 55). The short time required to administer the NVS (n = 51) resulted in more responses for this than for the TOFHLA (n = 47), which had the least number of participant completions. Interestingly, the mean score for the TOFHLA (86 overall combined mean score after weighting the numerical scale) places the group’s health literacy in the excellent range for this instrument, while the mean score of just under three for the NVS places the group’s health literacy in the ‘possible limited literacy’ range. Group mean scores for the HLS-EU-47 indicate that participants were strongest in items on the Healthcare scale while they were weakest in the Health promotion scale. The HLQ mean scores indicate a range of strengths and challenges for participants, with participants strongest in 1. Feeling understood and supported by health care providers; 4. Social support for health, 6. Ability to actively engage with health care providers; and 9. Understand health information well enough to know what to do, while people experience challenges in scales 3. Actively managing my health and 5. Appraisal of health information. The TOFHLA Reading comprehension scale exhibited a negative skewness (−2.15) and a high kurtosis (7.71), indicating a left skew where most scores are concentrated towards the higher end. Similarly, the TOFHLA Numeracy scale also demonstrated a negative left skew (−1.67) and kurtosis (6.12). In contrast, the NVS demonstrated near-zero skewness (0.03) and moderate kurtosis (6.12), which points to a symmetrical distribution of scores. The HLQ scales all have slight negative skewness, except for 7. Navigating the healthcare system, which has a skewness of 0.00, indicating a symmetrical distribution. The data skewness is reflected in the ceiling effects, which ranged from 11% to 68%. Apart from the NVS, all scales demonstrated some ceiling effects with more than 15% of the sample located in the top 15% of the scale range. The TOFHLA Reading comprehension and Numeracy scales showed the largest ceiling effect, with 68% and 62% of individuals scoring in the top 15% of the scale range, respectively. The NVS was the only scale demonstrating floor effects (18%).

We were interested in understanding the correlation between scales with content aiming to measure an individual’s functional health literacy. Specifically, we were interested in correlations between the TOFHLA, NVS, the HLS-EU-Q47 Healthcare scale and the HLQ scales 5. Appraisal of health information, 8. Ability to find good health information, and 9. Understand health information well enough to know what to do. There was a moderate correlation between both scales of the TOFHLA (Numeracy and Reading comprehension scales) and the NVS (r = 0.60 for both). There was no substantial correlations between any of the scales that used performance-based tests to assess functional health literacy (TOFHLA and NVS), and the multidimensional self-report instruments (HLS-EU-Q57 and the HLQ).

There was weak correlation between the HLS-EU-Q47 Healthcare scale and HLQ 1. Feeling understood and supported by healthcare providers. With the exception of the HLQ scale 3. Actively managing my health, there was low to moderate correlations between the HLS-EU-Q47 Healthcare scale and all other HLQ scales (r = 0.32 to 0.57), indicating that these scales capture somewhat related constructs. The highest correlation was r = 0.89 and was seen between two scales of the HLQ, 6. Ability to actively engage with healthcare providers and 7. Navigating the healthcare system.

Discussion

In this study we compared four commonly used health literacy instruments. Each of them aimed to measure functional health literacy in some way, however, there was only low to moderate correlation between instruments. In particular, the scales that aimed to measure functional health literacy (both of the TOFHLA scales, the NVS, the HLS-EU-Q47 Healthcare scale, and the HLQ scale 9. Understand health information well enough to know what to do) did not demonstrate any strong correlations. The TOFHLA was only moderately correlated with the NVS, while the HLS-EU-Q47 Healthcare scale demonstrated low to moderate correlations across all HLQ scales except scale 3.

There may be several methodological and conceptual explanations for the finding of little to no correlation between these scales. In terms of methodological explanations, these instruments were designed in different geographical contexts (HLS-EU-Q47 was designed in Europe, the HLQ in Australia, and the NVS and TOFHLA in the US). Health literacy is influenced by cultural and contextual factors, and some instruments may be more culturally sensitive and relevant to certain populations, which may lead to better performance on those scales (Warnecke et al. 1997). In addition, the response format and scale design of each instrument was different: even the HLQ and HLS-EU-Q47, which both use Likert scales, had slightly different response options that may have yielded different results, even when questions appear similar.

In terms of conceptual explanations, a potential reason for the low correlation between the performance-based functional health literacy instruments (NVS and TOFHLA) and the scales that measure functional health literacy on the multidimensional self-report instruments (HLS-EU-Q47 Healthcare scale, and the HLQ scale 9. Understand health information well enough to know what to do) may relate to the different approaches used to assess functional health literacy. The TOFHLA and NVS employ direct testing methods, assessing participants’ actual ability to read and understand information. In contrast, the HLS-EU-Q47 and HLQ rely on self-report of individuals’ experiences and abilities, including their perceived difficulty in performing specific health-related tasks and understanding medical forms. The use of self-report tools for measuring functional health literacy has some potential limitations when compared with performance measures. Respondents may not be fully aware of their own skills and abilities, leading to inflated or deflated self-reported views of competency (Hawkins et al. 2017), a phenomenon that has previously been described (Mabe and West 1982). Despite these limitations, self-reported assessments offer important benefits. They allow respondents to consider other factors that might influence their health literacy, such as social support received from a spouse or caregiver. For example, the HLQ scales enable clinicians or researchers to understand the role of social support in helping individuals comprehend and utilise health information effectively. People with limited functional health literacy may rely on caregivers as surrogate readers, compensating for their own deficiencies (Baker et al. 2002). While performance-based instruments provide direct insights into an individuals’ skills, self-report instruments offer a broader perspective, considering other factors that may mitigate the impact of limited functional health literacy.

The finding of skewness and ceiling effects were common across the HLQ, HLS-EU-Q47, and TOFHLA. These instruments may not discriminate well between people with high levels of health literacy but may be more likely to detect small differences between people with low and very low levels of health literacy, which is important for understanding and responding equitably to people experiencing health literacy challenges (WHO report; Osborne et al. 2022b). While the NVS was more evenly distributed, there were floor effects, which was similar to findings in other studies (Salgado et al. 2013; Tseng et al. 2018). This suggests that the NVS may not be able to discriminate between people with low to very low levels of literacy. The finding of moderate skew and ceiling effects for the HLQ have also been previously reported (Beauchamp et al. 2015; Rademakers et al. 2020) as well as skew and ceiling effects for the HLS-EU-Q47 (Duong et al. 2017; Pelikan et al. 2019). We could find no evidence of other studies reporting that the TOFHLA has high ceiling effects. It is possible that the study sample had a higher-than-average health literacy, which could contribute to higher score ranges than might be seen in a general population sample.

The high CR for the HLS-EU-47 Health promotion scale suggests there may be some item redundancy, which is likely to have been addressed with the more recent short versions (16 items in total) of the instrument. The high correlation between HLQ scales 6. Active engagement with healthcare providers and 7. Navigating the Healthcare system have previously been investigated, with findings concluding that at least two sources of reliable and substantive common factor variance may be represented: variance associated with the scale domains and variance associated with a general health literacy factor (Elsworth et al. 2022). Despite some evidence of correlation between some HLQ scales, the Elsworth study results support the current scoring of nine separate scales.

Conclusion

Improving individual and population health literacy is now a national priority for many countries. This study found that the TOFHLA, NVS, HLS-EU, and HLQ are only moderately or weakly associated with each other, indicating that while they are all measures of health literacy, they do measure different things. Therefore when choosing an instrument, clinicians and researchers need to be cognisant of their intended measurement purpose (what sorts of decisions will be made based on score interpretations) and the types of health literacy constructs the different instruments measure.

Supplementary material

Supplementary material is available online.