Validity of four measures in assessing school canteen menu compliance with state-based healthy canteen policy
Kathryn Reilly A B C D E , Nicole Nathan A B C D , Luke Wolfenden A B C D , John Wiggers A B C D , Rachel Sutherland A B C D , Rebecca Wyse A B C D and Sze Lin Yoong A B C D
+ Author Affiliations
- Author Affiliations
A School of Medicine and Public Health, University of Newcastle, University Drive, Callaghan, NSW 2308, Australia.
B Hunter Medical Research Institute, Locked Bag 1000, New Lambton, NSW 2305, Australia.
C Priority Research Centre for Health Behaviour, University of Newcastle, University Drive, Callaghan, NSW 2308, Australia.
D Hunter New England Population Health, Locked Bag 10, Wallsend, NSW 2287, Australia.
E Corresponding author. Email: kathryn.reilly@hnehealth.nsw.gov.au
Health Promotion Journal of Australia 27(3) 215-221 https://doi.org/10.1071/HE16053
Submitted: 11 May 2016 Accepted: 4 August 2016 Published: 16 September 2016
Journal Compilation © Australian Health Promotion Association 2016
Abstract
Issue addressed: In order to assess the impact of healthy school canteen policies on food availability for students, valid methods of measuring compliance are needed that can be applied at scale. The aim of this study is to assess the validity and direct cost of four methods to assess policy compliance: 1) principal and 2) canteen manager self-report via a computer-assisted telephone interview; and 3) comprehensive and 4) quick menu audits by dietitians, compared with observations.
Methods: A cross-sectional study took place in the Hunter region of NSW, Australia, in a sample of 38 primary schools that had previously participated in a randomised controlled trial to improve healthy canteen policy compliance. Policy compliance was assessed using the four methods specified above. Percentage agreement, kappa, sensitivity and specificity compared with observations was calculated together with the direct time taken and costs of each method. Indirect costs (including set-up costs) for all measures have not been included.
Results: Agreement with observations was substantial for the quick menu audit (kappa = 0.68), and moderate for the comprehensive menu audit (kappa = 0.42). Principal and canteen manager self-report resulted in poor agreement and low specificity with the gold standard. The self-reported measures had the lowest cost, followed by the quick menu audit and lastly the comprehensive menu audit.
Conclusion: The quick menu audit represents a valid and potentially low-cost method of supporting policy implementation at scale.
So what?: This study demonstrates that a quick menu audit represents a valid measure of undertaking assessment of school canteen policy compliance at a population level.
Key words: children, evaluation methods, health policy, health-promoting schools, nutrition.
References
[1] Kaikkonen JE, Mikkila V, Magnussen CG, Juonala M, Viikari JS, Raitakari OT (2013) Does childhood nutrition influence adult cardiovascular disease risk? – Insights from the Young Finns Study. Ann Med 45, 120–8.| Does childhood nutrition influence adult cardiovascular disease risk? – Insights from the Young Finns Study.Crossref | GoogleScholarGoogle Scholar | 22494087PubMed |
[2] Shay CM, Gooding HS, Murillo R, Foraker R (2015) Understanding and improving cardiovascular health: an update on the American Heart Association’s concept of cardiovascular health. Prog Cardiovasc Dis 58, 41–9.
| Understanding and improving cardiovascular health: an update on the American Heart Association’s concept of cardiovascular health.Crossref | GoogleScholarGoogle Scholar | 25958016PubMed |
[3] Krebs-Smith SM, Guenther PM, Subar AF, Kirkpatrick SI, Dodd KW (2010) Americans do not meet federal dietary recommendations. J Nutr 140, 1832–8.
| Americans do not meet federal dietary recommendations.Crossref | GoogleScholarGoogle Scholar | 1:CAS:528:DC%2BC3cXhsFSnurzF&md5=3521933ba296fa79807066a37d1912cfCAS | 20702750PubMed |
[4] Public Health England. National diet and nutrition survey. Results from years 1–4 (combined) of the rolling program (2008/2009–2011/2012). 2014. Available from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/310997/NDNS_Y1_to_4_UK_report_Executive_summary.pdf [Verified 8 August 2016].
[5] Whitrow MJ, Moran L, Davies MJ, Collins CE, Burrows TL, Edwards S, et al (2016) Core food intakes of Australian children aged 9–10 years: nutrients, daily servings and diet quality in a community cross-sectional sample. J Hum Nutr Diet
| Core food intakes of Australian children aged 9–10 years: nutrients, daily servings and diet quality in a community cross-sectional sample.Crossref | GoogleScholarGoogle Scholar | 27028485PubMed |
[6] Mikkilä V, Räsänen L, Raitakari OT, Pietinen P, Viikari J (2005) Consistent dietary patterns identified from childhood to adulthood: the Cardiovascular Risk in Young Finns Study. Br J Nutr 93, 923–31.
| Consistent dietary patterns identified from childhood to adulthood: the Cardiovascular Risk in Young Finns Study.Crossref | GoogleScholarGoogle Scholar | 16022763PubMed |
[7] Hawkes C, Smith TG, Jewell J, Wardle J, Hammond RA, Friel S, et al (2015) Smart food policies for obesity prevention. Lancet 385, 2410–21.
| Smart food policies for obesity prevention.Crossref | GoogleScholarGoogle Scholar | 25703109PubMed |
[8] Story M, Kaphingst KM, Robinson-O’Brien R, Glanz K (2008) Creating healthy food and eating environments: policy and environmental approaches. Annu Rev Public Health 29, 253–72.
| Creating healthy food and eating environments: policy and environmental approaches.Crossref | GoogleScholarGoogle Scholar | 18031223PubMed |
[9] Bell AC, Swinburn BA (2004) What are the key food groups to target for preventing obesity and improving nutrition in schools? Eur J Clin Nutr 58, 258–63.
| What are the key food groups to target for preventing obesity and improving nutrition in schools?Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BD2c%2FkvVChsg%3D%3D&md5=30ffa51431c0753edc14e1b20aa76822CAS | 14749745PubMed |
[10] Sanigorski AM, Bell AC, Kremer PJ, Swinburn BA (2005) Lunchbox contents of Australian school children: room for improvement. Eur J Clin Nutr 59, 1310–6.
| Lunchbox contents of Australian school children: room for improvement.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BD2MrosV2msw%3D%3D&md5=3bc3ba895749f7d631706e0e868dc61bCAS | 16034359PubMed |
[11] WHO. Global action plan for the prevention and management of noncommunicable diseases 2013–2020. Geneva: WHO Document Production Services; 2013.
[12] L’Abbé M, Schermel A, Minaker L, Kelly B, Lee A, Vandenijvere S, et al (2013) Monitoring foods and beverages provided and sold in public sector. Obes Rev 14, 96–107.
| Monitoring foods and beverages provided and sold in public sector.Crossref | GoogleScholarGoogle Scholar | 24074214PubMed |
[13] Woods J, Bressan A, Langelaan C, Mallon A, Palermo C (2014) Australian school canteens: menu guideline adherence or avoidance? Health Promot J Austr 25, 110–5.
| Australian school canteens: menu guideline adherence or avoidance?Crossref | GoogleScholarGoogle Scholar | 25200467PubMed |
[14] NSW Department of Health (DoH) Department of Education and Training. (DET). Fresh Tastes @ School NSW Healthy School Canteen Strategy: canteen menu planning guide. 3rd edn. Sydney: NSW DoH, NSW DET; 2012. Available from: https://www.det.nsw.edu.au/policies/student_serv/student_health/nutrition/implementation_1_PD20110420.shtml [Verified 16 August 2016].
[15] Ardzejewska K, Tadros R, Baxter D (2013) A descriptive study on the barriers and facilitators to implementation of the NSW (Australia) Healthy School Canteen Strategy. Health Educ J 72, 136–45.
| A descriptive study on the barriers and facilitators to implementation of the NSW (Australia) Healthy School Canteen Strategy.Crossref | GoogleScholarGoogle Scholar |
[16] de Silva-Sanigorski A, Breheny T, Jones L, Lacy K, Kremer P, Carpenter L, et al (2011) Government food service policies and guidelines do not create healthy school canteens. Aust NZ J Publ Health 35, 117–21.
| Government food service policies and guidelines do not create healthy school canteens.Crossref | GoogleScholarGoogle Scholar |
[17] Dick M, Lee A, Bright M, Turner K, Edwards R, Dawson J, et al (2012) Evaluation of implementation of a healthy food and drink supply strategy throughout the whole school environment in Queensland state schools, Australia. Eur J Clin Nutr 66, 1124–9.
| Evaluation of implementation of a healthy food and drink supply strategy throughout the whole school environment in Queensland state schools, Australia.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC38fnvFyruw%3D%3D&md5=7c227c68942972bad6b17e65e4ea0642CAS | 22892436PubMed |
[18] Hills A, Nathan N, Robinson K, Fox D, Wolfenden L (2015) Improvement in primary school adherence to the NSW Healthy School Canteen Strategy in 2007 and 2010. Health Promot J Austr 26, 89–92.
| Improvement in primary school adherence to the NSW Healthy School Canteen Strategy in 2007 and 2010.Crossref | GoogleScholarGoogle Scholar | 25993892PubMed |
[19] Pettigrew S, Donovan RJ, Jalleh G, Pescud M (2014) Predictors of positive outcomes of a school food provision policy in Australia. Health Promot J Int 29, 317–27.
| Predictors of positive outcomes of a school food provision policy in Australia.Crossref | GoogleScholarGoogle Scholar |
[20] Lawlis T, Knox M, Jamieson M (2016) School canteens: a systematic review of the policy, perceptions and use from an Australian perspective. Nutr Diet
| School canteens: a systematic review of the policy, perceptions and use from an Australian perspective.Crossref | GoogleScholarGoogle Scholar |
[21] McGraw SA, Sellers D, Stone E, Resnicow KA, Kuester S, Fridinger F, et al (2000) Measuring implementation of school programs and policies to promote healthy eating and physical activity among youth. Prev Med 31, S86–97.
| Measuring implementation of school programs and policies to promote healthy eating and physical activity among youth.Crossref | GoogleScholarGoogle Scholar |
[22] McPherson RS, Hoelscher DM, Alexander M, Scanlon KS, Serdula MK (2000) Dietary assessment methods among school-aged children: validity and reliability. Prev Med 31, S11–33.
| Dietary assessment methods among school-aged children: validity and reliability.Crossref | GoogleScholarGoogle Scholar |
[23] Adams SA, Matthews C, Ebbeling C, Moore C, Cunningham J, Fulton J, et al (2005) The effect of social desirability and social approval on self report of physical activity. Am J Epidemiol 161, 389–98.
| The effect of social desirability and social approval on self report of physical activity.Crossref | GoogleScholarGoogle Scholar | 15692083PubMed |
[24] Australian Bureau of Statistics. Census Data. Canberra: Commonwealth of Australia; 2011. Available from: http://www.censusdata.abs.gov.au/census_services/getproduct/census/2011/quickstat/111?opendocument&navpos=220 [Verified 16 August 2016].
[25] Wolfenden L, Nathan N, Williams C, Delaney T, Reilly K, Freund M, et al (2014) A randomised controlled trial of an intervention to increase the implementation of a healthy canteen policy in Australian primary schools: study protocol. Implement Sci 9, 147
| A randomised controlled trial of an intervention to increase the implementation of a healthy canteen policy in Australian primary schools: study protocol.Crossref | GoogleScholarGoogle Scholar | 25300221PubMed |
[26] National Health and Medical Research Council. Australian Dietary Guidelines. Canberra: National Health and Medical Research Council; 2013.
[27] NSW Department of Health. Good for kids. Good for life; Canteen product database. 2016. Available from: http://www.goodforkids.nsw.gov.au/primary-schools/canteens/product-search/ [Verified 16 August 2016].
[28] Yoong SL, Nathan NK, Wyse RJ, Preece SJ, Williams CM, Sutherland RL, et al (2015) Assessment of the school nutrition environment: a study in Australian primary school canteens. Am J Prev Med 49, 215–22.
| Assessment of the school nutrition environment: a study in Australian primary school canteens.Crossref | GoogleScholarGoogle Scholar | 26091931PubMed |
[29] Australian Bureau of Statistics. Technical paper: census of population and housing: Socioeconomic Indexes For Australia (SEIFA). Cat. no 2039.0.55.001.Canberra: Commonwealth of Australia; 2006. Available from: http://www.abs.gov.au/ausstats/abs@.nsf/mf/2039.0.55.001 [Verified 16 August 2016].
[30] Australian Bureau of Statistics. Australian Standard Geographical Classification (ASGC). Cat. no 1216.0. Canberra: Commonwealth of Australia; 2011. Available from: http://www.abs.gov.au/ausstats/abs@.nsf/mf/1216.0 [Verified 16 August 2016].
[31] New South Wales Department of Education and Training. 2009 DET Directory. Sydney, NSW: Department of Education and Training; 2009.
[32] Hartmann DP (1977) Considerations in choice of interobserver reliability estimates. J Appl Behav Anal 10, 103–16.
| Considerations in choice of interobserver reliability estimates.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DC%2BC3cnitFyksg%3D%3D&md5=bbc03038dae8754abc5744ce9b4a6715CAS | 16795538PubMed |
[33] Byrt T, Bishop J, Carlin J (1993) Bias prevalence and kappa. J Clin Epidemiol 46, 423–9.
| Bias prevalence and kappa.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DyaK3s3nsVejsw%3D%3D&md5=8e1e77c96710c7942cebb463904593f1CAS | 8501467PubMed |
[34] Landis JR, Kotch G (1977) The measurement of observer agreement for categorical data. Biometrics 33, 159–74.
| The measurement of observer agreement for categorical data.Crossref | GoogleScholarGoogle Scholar | 1:STN:280:DyaE2s7jsFWqtA%3D%3D&md5=0b1245c6d4e28d81a498f7aa155bad8bCAS | 843571PubMed |
[35] Dietitians Association of Australia. Private practice fee survey results. 2009. Available from: http://dmsweb.daa.asn.au/files/DAA_A_Z/R_V/Private_Practice_%20Fee_Survey_Results.pdf [Verified 16 August 2016].
[36] Nathan N, Wolfenden L, Morgan PJ, Bell AC, Barker D, Wiggers J (2013) Validity of a self-report survey tool measuring the nutrition and physical activity environment of primary schools. Int J Behav Nutr Phys Act 10, 75
| Validity of a self-report survey tool measuring the nutrition and physical activity environment of primary schools.Crossref | GoogleScholarGoogle Scholar | 23758936PubMed |
[37] Manske S. In pilot phase of the 2007–2008 school health environment survey: technical report. Waterloo, Ontario: University of Waterloo; 2008.
[38] Wolfenden L, Kypri K, Britton B, James EL, Francis JL, Wyse R (2013) Effects of introductory information on self-reported health behavior. Epidemiology 24, 170–2.
| Effects of introductory information on self-reported health behavior.Crossref | GoogleScholarGoogle Scholar | 23232620PubMed |
[39] WASCA Inc. Star choice program product database 2016. Available from: http://www.waschoolcanteens.org.au/star-choice-program/product-database/ [Verified 16 August 2016].
[40] Nathan N, Wolfenden L, Bell A, Wyse R, Morgan P, Butler M, et al (2012) Effectiveness of a multi-strategy intervention in increasing the implementation of vegetable and fruit breaks by Australian primary schools: a non-randomized controlled trial. BMC Public Health 12, 651
| Effectiveness of a multi-strategy intervention in increasing the implementation of vegetable and fruit breaks by Australian primary schools: a non-randomized controlled trial.Crossref | GoogleScholarGoogle Scholar | 22889085PubMed |
