Validity of four measures in assessing school canteen menu compliance with state-based healthy canteen policy

Key words: children, evaluation methods, health policy, health-promoting schools, nutrition.

Suboptimal dietary intake, including excess kilojoule intake and low fruit and vegetable consumption, is associated with overweight, obesity and chronic diseases including cardiovascular disease and some types of cancer.^1,2 In countries such as the United States (USA), the United Kingdom (UK) and Australia, it has been reported that ~90% of adults and children do not consume adequate vegetables and fruit to meet nationally recommended guidelines, and that the majority consume foods high in energy, sodium, saturated fat and/or sugar on a daily basis.^3–5 As eating behaviours and habits formed during childhood persist into adulthood,⁶ interventions to improve child dietary intake are recommended as a key strategy in reducing the future burden of chronic disease.⁷

Schools are recommended as a relevant setting to improve children’s dietary intake as they provide access to almost all children during a key developmental period.⁸ Importantly, children can consume a significant proportion (almost 40%) of their dietary intake while at school.⁹ In Australia, in the majority of schools, children can purchase foods and drinks from a canteen or tuckshop.¹⁰ With over 7000 school canteens in Australia, they represent one of the largest and most frequently accessed food outlets for school-aged children.⁹

To support schools implement strategies to improve the nutrition of children, the World Health Organization’s Global Action Plan encourages governments to develop or strengthen national food and nutrition policies and action plans in public institutions including schools.¹¹

A further recommendation is that the implementation of such strategies be monitored and evaluated to ensure such programs are effective.¹¹ Internationally there have been concerted efforts to support the monitoring of nutrition environments. The International Network for Food and Obesity/Non-communicable Diseases Research, Monitoring and Action Support group (INFORMAS), a global network of public interest organisations and researchers, have outlined a framework for monitoring the provision of food in line with nutrition policies in public settings including schools.¹² The group outlines a stepwise framework for monitoring, which includes a variety of approaches including ‘direct observations or on-site visits’ as the optimal approach, or the use of menu audits and self-report as approaches to obtain data from large numbers of schools.¹²

In Australia, all states and territories have introduced healthy canteen policies that utilise a traffic light system to promote healthy foods and restrict the sale of less healthy foods.¹³ In NSW in 2005, the government mandated a healthy school canteen policy – Fresh Tastes @ School (FT@S) Healthy Canteen Strategy – for all government primary and secondary schools.¹⁴ The guidelines are based on best-practice dietary guidelines at the time¹⁴ and use a traffic light system to categorise menu items based on nutritional profile. ‘Green’ menu items are sources of essential nutrients, contain less saturated fat and/or added sugar and/or salt. ‘Amber’ foods are mainly processed foods with some nutritional value that can, in large serve sizes, contribute to excess energy intake.¹⁴ ‘Red’ foods lack nutritional value, are high in saturated fat, and/or added sugar and/or salt, and can contribute to excess energy intake.¹⁴ To be compliant with the policy, school canteens are encouraged to fill the menu with ‘green’ foods and restrict the sale of ‘amber’ and ‘red’ foods. In 2007, a ban on sales of sugar-sweetened drinks with more than 300 kJ per serve or more than 100 mg sodium per serve was introduced.¹⁴ The FT@S guidelines provide a Ready Reckoner¹⁴ of ‘green’, ‘amber’ or ‘red’ commonly sold foods in school canteens. Other menu items, including most commercial products, require comparison to the Occasional Food Criteria Table, which provides specific nutrient cut-off points for kilojoules, saturated fat, sodium and fibre. For this purpose, additional detail regarding menu products – such as brand, serve size and flavour – are needed to allow for classification according to the policy. Typically, schools offer two types of menus during the school year: a ‘summer menu’ during the warmer months and a ‘winter menu’ during the colder months.

Despite the popularity of such policies in Australia, only a small number of studies have assessed whether schools adequately implement these policies. These studies have found variable implementation ranging from 0–97%,^13–20 which may be due to the use of different methods of measuring policy implementation across the studies. Studies that use principal or canteen manager self-report typically report a higher compliance rate (61–97%)^17,19 compared with studies where menu audits were undertaken (0–62%).^14,16,18 One study that undertook observations of food items within four schools found that none was compliant with state policy guidelines.¹⁵

To assess the impact of healthy canteen policies on food provision in schools, valid tools that are inexpensive and time-efficient in their administration are needed to assess policy compliance at a population level.²¹ While observations are the ‘Gold Standard’ method for assessing school nutrition environments,²² this method is typically costly to administer as it relies on on-site observations by trained field staff.²¹ As such, it is impractical for ongoing monitoring of entire school populations on a jurisdictional basis. Menu audits can be conducted with canteen managers via telephone or email and represent a potentially less expensive method with greater reach. This process, however, requires dietary assessment expertise in auditing the menus¹⁶ and relies on canteen managers knowing the nutritional profile of products sold in the canteen to accurately classify foods. Brief self-reported measures represent the lowest cost approach and the one most readily administered. These measures, however, have been suggested to overestimate policy compliance.²³ The relative validity and cost of these policy compliance assessment methods have not been reported previously. This represents a significant impediment to research aimed at improving or monitoring policy compliance.

This study compares the relative validity and cost of four school canteen policy compliance assessment methods: (1) principal self-report, (2) canteen manager self-report, (3) comprehensive menu audit, and (4) quick menu audit with observations. Further, the direct cost and time of undertaking each canteen policy compliance assessment method was also described.

Approval to conduct the study was obtained from Hunter New England Area Health Service Human Research Ethics Committee (no. 06/07/26/4.04), the University of Newcastle (H-2008–0343), the New South Wales Department of Education (DoE); and relevant Catholic School Offices.

This cross-sectional study compared four methods of assessing school canteen menu compliance with the state healthy canteen policy with observations. The study took place in the Hunter region of NSW, which has a socioeconomically and demographically diverse population of ~74 709 children aged 5–14 years.²⁴

Out of 159 public primary schools (servicing children aged 5–12 years) in the Hunter region that had an operational canteen,70 were randomly selected to participate in a randomised controlled trial (RCT) undertaken by the research team.²⁵ From these schools a quota sample of 50 were invited to participate in this validation study.

From October–December 2014 (summer menu, also offered during February–April), principals at all primary schools in the Hunter region of NSW were approached by letter to participate in a computer-assisted telephone interview (CATI) regarding the promotion of healthy eating and physical activity in schools. Principals were telephoned 2 weeks later by a trained interviewer who confirmed school eligibility, sought consent and scheduled a time to complete the CATI. Consent was also sought from principals to invite canteen managers to participate in a separate CATI. Where such consent was obtained, an information letter was sent to the canteen manger and followed up with a telephone call to participate in the CATI.

Canteen managers were asked to provide a copy of their current (summer) canteen menus to the research team on the day of observations (February–April) for audit by a dietitian using both the comprehensive and quick menu audit methods.

A subsample (n = 50) of principals and canteen managers was asked to provide consent for observations of canteen food and beverage products. A research assistant subsequently contacted the schools to arrange a suitable time to undertake the observations.

During the CATI, principals were asked the number of students attending the school. School postcode was also obtained from school websites.

Principals were asked: ‘Does your school provide healthy food options consistent with the FT@S menu guidelines in the canteen?’(Yes/No/Don’t know).

Canteen managers were asked: ‘Is your canteen currently compliant with FT@S?’ (Yes/No/Not sure).

Both principal and canteen manager’s one-item measure was embedded in a larger survey of school healthy eating and physical activity practices.

Comprehensive menu audits were completed by trained dietitians with extensive knowledge of the FT@S guidelines and experience in carrying out audits of school canteen menus. A standardised Menu Assessment Protocol was developed based on the FT@S guidelines and the Australian Dietary Guidelines for Children and Adolescents,²⁶ which outlines the menu assessment procedure including a step-by-step process for collecting additional product information and colour coding menu items according to the FT@S guidelines. The protocol also includes an assumptions list for menu items where the colour code has not been clearly defined in the FT@S guidelines and menu counting guidelines. A menu audit was undertaken using a canteen product database of most commonly sold canteen products in the Hunter New England region developed by the research team.²⁷ This database was developed based on the team’s experience working with school canteens, collection of nutritional information of products provided by local suppliers to school canteens in the region and the assessment of menus from over 200 schools. Additionally, canteen managers in each school were telephoned to collect additional information about a food or beverage item such as product brand, serve size or flavours not typically provided on menus. Dietitians used a standard template to record any additional information needed to assess compliance. On average three phone calls were required per school to collect this additional information. All menu items were colour coded according to the FT@S guidelines as ‘green’, ‘amber’ or ‘red’. A double audit of 15 menus using the comprehensive menu audit by two independent dietitians achieved a high percentage of agreement (90%) in relation to the percentage of ‘green’ and ‘red’ menu items.²⁸

The quick menu audit method was adapted from the comprehensive menu audit by a team of dietitians, using the following steps: (1) assessment of canteen products to develop a centralised database of the most commonly sold canteen products in the HNE region;²⁷ (2) engagement with key stakeholders, which included communication with canteen managers, suppliers and health promotion practitioners supporting schools; (3) piloting of the quick menu audit tool – the measure was pilot tested using different assumptions needed to classify items according to the FT@S criteria where product information is not available (e.g. full-fat vs low-fat dairy) and modified accordingly; and (4) evidence-based application of policy/guidelines similar to the comprehensive menu audit. Based on this, the tool assigns product information and serve sizes for each menu item, eliminating the need to collect such additional information from canteen managers. The tool consists of a detailed list of common canteen menu items grouped into categories such as drinks, hot food, frozen dairy treats, snacks, sandwiches and salads, with colour-coded classifications and justifications for each assumption made. Two trained dietitians independently carried out double audits of the 38 schools using the quick menu audit method and found a 100% agreement in terms of having a menu that was compliant to the FT@S policy. All menu items were colour coded according to the FT@S guidelines as ‘green’, ‘amber’ or ‘red’.

Two to three research assistants collected observational data regarding all food and beverages sold in schools on a single day of data collection. The observations were conducted between February and April 2015 (summer menu). On the day of data collection, research assistants recorded nutritional information from product nutrition panels of all food and beverage items sold in the canteen to classify items according to the FT@S guidelines. For food products made by canteen staff (e.g. sandwiches), recipes were obtained from the canteen manager and the nutrient profile of included food items was generated using a nutrient analysis software package (FoodWorks). A menu audit was carried out using the collected data to classify menu items according to the FT@S guidelines (‘green’, ‘amber’ or ‘red’).

The research assistants underwent a day’s training in recording product nutritional information and use of audit tools by a dietitian. Data collection tools were developed and piloted in two school canteens before their use. Training incorporated quality assurance tests with all research assistants required to score 100% agreement with a dietitian before commencement of data collection.

For time and cost estimation, only direct costs (e.g. salary and time taken to directly administer the measures) related to obtaining information needed to assess compliance were included. All indirect costs were excluded from cost estimations. For self-reported measures, indirect costs related to development of the questionnaire and programming of the CATI were excluded. The cost of undertaking the telephone calls was also excluded. For menu audits, indirect costs related to development of the canteen database, interviews and pilot testing with stakeholders in the field were excluded. Further, time taken to collect school menus was not included as menus were collected during observations. For both principal and canteen manger self-report, the one-item measure described here was conducted as part of a larger survey, which took 20–30 min to complete. The cost per completed survey is described in the manuscript, as it was not possible to isolate the cost for a single question. For the comprehensive and quick menu audit, staff time taken for collection of additional information and completion of menu audits were calculated where relevant.

Statistical analyses were conducted using SAS version 9.3 (SAS Institute Inc., Cary, NC, USA). Descriptive statistics were used to describe the characteristics of participating schools. School postcodes were used to categorise schools into ‘higher’ and ‘lower socioeconomic’ regions using the Socioeconomic Indexes For Australia (SEIFA) database.²⁹ School postcodes were also used to categorise schools as rural (outer regional, remote, and very remote areas) or urban (major cities and inner regional areas) using the Accessibility/Remoteness Index of Australia (ARIA).³⁰ Schools were categorised as small (1–159 students); medium (160–450 students); or large (451+ students) based on number of students enrolled.³¹

For menu audits, total menu items were tallied and percentage of ‘green’, ‘amber’ and ‘red’ items was calculated. Schools were classified as compliant with the policy if the menu did not contain foods or beverages restricted from regular sale (‘red’ and banned items) and had healthy items (‘green’ items) representing the majority (> 50%) of products. For self-report measures, principal and canteen manager responses of ‘yes’ to the survey question were deemed compliant. Percentage agreement, sensitivity, specificity, predictive and kappa values were reported for each of the four measurement methods compared with observations. Percentage agreement of 80% or greater was considered ‘strong agreement’.³² In order to take into account agreement by chance, kappa is reported in addition to percent agreement. Consistent with previous research,³³ where positive agreement accounted for over 75% or under 25% of total agreement, prevalence-adjusted and bias-adjusted kappa (PABAK) was reported based on benchmarks suggested by Landis and Koch (< 0.00 = poor, 0.00–0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial, 0.81–1.0 = almost perfect).³⁴ The sensitivity, specificity, positive predictive values (PPV) and negative predictive values (NPV) of all four measures relative to the observational audits were calculated with 95% confidence intervals. Sensitivity is the proportion of schools found to be compliant with the FT@S policy that were identified as compliant through observations. Specificity is the proportion of schools found to be non-compliant that were identified as non-compliant through observations.

The direct cost of the principal and canteen manager self-report was calculated using the total casual salary cost (A$13 805.23 and A$6939.79 respectively) of each CATI divided by the number of schools surveyed. The cost per menu audit was calculated using the average hourly rate of A$113/h for menu audit by dietitians in private practice, according to a Dietitians Association of Australia survey (2009).³⁵

Consent to participate in observations was given by 38 of the 50 schools approached to participate (78% consent rate). Of these 38 schools, 58% were classified as higher socioeconomic status, 82% were located in a major city and the majority of schools were medium-sized (63%) (see Table 1). There was no significant differences between the included sample and rest of the primary schools located in the Hunter region in terms of school size (student enrolment: small, medium, large) (P = 0.21), socioeconomic region (P = 0.18) or remoteness (P = 0.29). Twenty-six canteen managers and 30 principals answered the CATI question regarding menu compliance with the FT@S policy.

Table 1.  Descriptive statistics of study sample

Observations found 16% of the 38 schools (n = 6) had menus compliant with the FT@S guidelines. The quick menu audit produced the highest percentage agreement (84%) and kappa rating (k = 0.68) with observational audits, followed by the comprehensive menu audit (71% agreement, k = 0.42) (see Table 2). Likewise, the quick menu audit had the highest PPV and NPVs, followed by the comprehensive menu audit. Both principal and canteen manager self-report on compliance resulted in 100% sensitivity, but poor percentage agreement, kappa, specificity, and positive and negative predictive values with observations.

Table 2.  Percentage agreement, sensitivity, specificity, positive predictive values, negative predictive values, kappa in canteen menu compliance based on FT@S guidelines (n = 38)
CI, confidence interval

Table 3 presents the average time taken and corresponding staff costs associated with the four measures of compliance. The quick menu audit method took on average 10 min per menu, costing approximately $18.83. The comprehensive menu audit took on average 45 min to complete at just under $85.00 per menu. Principal and canteen manager self-reports cost $34.17 and $26.29, respectively; however, this accounted for the entire CATI (58 and 55 items in total, respectively) and not the single policy compliance measure alone.

Table 3.  Direct cost per measure of compliance per school
NA, menu audit not undertaken as part of this measure

This study sought to assess the validity of various measures of compliance, including varying levels of cost and time burden, to the NSW FT@S Healthy Canteen Strategy. The quick menu audit resulted in best agreement with observations. In contrast, both principal and canteen manager self-report had the lowest agreement (13% and 23%, respectively) and low specificity. These findings are similar to those of other studies that have found low agreement between principals/teachers and their ability to report on foods sold in the school canteen and vending machines relative to observations.^36,37 The comprehensive menu audit had lower agreement and sensitivity/specificity than the quick menu audit, despite significant investment to obtain additional information. As the comprehensive menu audit relies on additional reporting by canteen managers in regards to nutritional information of products sold in the canteen, this method may be limited by recall and social desirability bias associated with the reporting of such items.²³ Findings from this study indicate that self-reported one-item measures, while feasible and lower cost to administer, do not provide an accurate representation of policy compliance. Our study found that both principal and canteen manager self-report had very low specificity indicating inadequate assessment of non-compliance. Such findings are likely due to social desirability bias associated with self-report,²³ which can result in a significant overestimation of compliance. Future use of self-report measures should include strategies such as the inclusion of appropriate introductory information to minimise such reporting biases.³⁸

While the quick menu audit incurred low direct cost, the tool and product database was developed based on the research team’s extensive work with key stakeholders in the region and substantial investment in obtaining information about foods sold in canteens. This study did not quantify the costs and time required to establish this infrastructure. These costs are likely to be significant suggesting that future efforts to implement such an approach to measuring school canteen compliance are likely to require substantial investment to ensure the validity and applicability of the tool to a local level context. While some adaptation of the database is required, it is likely that product information of nationally available commercial products could be transferred across jurisdiction. Future efforts to monitor the impact of food service or healthy canteen policies should consider such costs when adapting menu audit processes to support local level evaluation. For example, the Western Australia School Canteen Association product database developed in conjunction with the National Heart Foundation and Department of Health³⁹ provides a potential infrastructure to support development of such menu audit tools for state-wide assessment of compliance with healthy canteen policies.

A strength of this study is the comparison of multiple measures of compliance to a gold standard measure. The inclusion of labour and time costs provides essential information regarding the utility of such measures to be applied at scale. A limitation of the study is the relatively small number (38) of schools included in the sample. Given the differences in food services found in schools internationally, use of the quick menu audit method is likely to be limited to regions that provide a canteen facility similar to that found in Australian schools. Similar to other studies assessing compliance in schools,¹⁹ the self-report measures only consisted of one item. While increasing the number of items to assess compliance may have increased specificity of this tool, a previous validation study in Australian schools suggest that principals can accurately report on the implementation of a fruit and vegetable project using a one-item measure.⁴⁰ This study also did not assess indirect costs (including development of the canteen database and set-up of the CATIs) associated with the measures. While the initial costs associated with development of the canteen database that underpins the menu audit methods are likely to be significant, ongoing costs in updating this database are likely to be minimal once established.

Findings from this study indicate that self-reported measures are unlikely to provide an accurate representation of policy compliance. The quick menu audit represents an inexpensive, relative to a gold standard approach, and valid method that can be used to assess healthy canteen policy compliance on a large scale. The availability of such valid measures are essential to support future research assessing the impact of intervention strategies to overcome policy implementation failure in this field.