Impact of targeted wording on response rates to a survey of general practitioners on referral processes for suspected head and neck cancer: an embedded randomised controlled trial

Keywords: diagnosis delay, general practice, geography, head and neck cancer, health-care access, primary care, randomised controlled trial, variation.

Response rates to surveys of general practitioners (GPs) vary greatly, ranging in the literature from 7 to 67%.^1–4 This can result in underpowered studies, raising concern for response bias and limit the generalisability of findings. Methods to increase response rates to surveys vary based on whether they are administered electronically or by hard copy,^5,6 but can include incentives, pre-notification, follow up, and personalised approaches. A meta-analysis conducted by Edwards et al.⁵ found that more interesting or salient questionnaires had double the response rate compared to less interesting or non-salient versions (OR 2.00, 95%CI 1.32, 3.04). This suggests that question and content should be considered when designing questionnaires.

Cancer is a relatively uncommon presentation in primary care. It is estimated that Australian GPs will see only four ‘serious’ cancer cases per year.⁷ For uncommon cancers, like head and neck cancer, presentations are far fewer. Data from Finland suggests that GPs may only see two new cases of head and neck cancer in their career,⁸ though some 11% of symptoms experienced by patients visiting a GP may also be experienced by patients with head and neck cancer. This raises the question as to whether GPs may self-exclude from surveys on head and neck cancer if they see these presentations infrequently in practice, and therefore presents a challenge to obtain an optimal sample of survey participants to ensure external validity of results and relevance to practice.

Though cancer diagnoses in primary care are rare, associated symptoms are relatively common, and so it is plausible that GPs may be more willing to participate in research that addresses the latter. Therefore, we conducted this study to investigate whether the framing of the study materials affected survey response rates. The aim of this study was to determine whether targeted wording impacted on response rates to a survey of Australian GPs. We hypothesised that response rates would be higher among GPs provided with a questionnaire written with a ‘symptom-frame’ compared to a ‘cancer-frame’.

This randomised controlled trial was embedded within a survey of GPs investigating referral processes for suspected head and neck cancer. The aim of the survey was to investigate geographical variation in self-reported management by GPs of patients with symptoms suggestive of head and neck cancer in the context of possible delayed referral for head and neck cancer in primary care.

GPs were identified from publicly available lists in two Primary Health Networks in New South Wales, Australia – the Central and Eastern Sydney Primary Health Network,⁹ and the North Coast Primary Health Network.¹⁰ These Primary Health Networks were selected as they mirrored the sampling area of patients who were being recruited for a related study of routes and times to diagnosis and treatment of head and neck cancer in New South Wales. GPs were eligible if they were a practicing GP within the Central and Eastern Sydney Primary Health Network or North Coast Primary Health Network. A total of 1875 unique GPs participants were identified (n = 1556 from the Central and Eastern Sydney Primary Health Network, n = 319 from the North Coast Primary Health Network) across 660 practices. GPs were excluded if they were a trainee, deceased, retired, unable to be contacted (i.e. return to sender with no forwarding address), on extended leave, or had moved out of the area.

Two versions of the same survey materials were written for the study, with minor variations based on the frame (Table 1). Group A (‘symptom-frame’) received a version of the study materials (advance notification flyer, invitation letter, participant information sheet, and questionnaire) using the term ‘red-flag upper aerodigestive symptoms’. Group B (‘cancer-frame’) received a version of the study materials using the term ‘suspected head and neck cancer’. Survey administration procedures were identical in both groups, with one advance notification flyer, followed by a mailed version of the questionnaire (with a hand-signed invitation letter, participant information sheet, and reply paid envelope). Up to three mailed reminders were sent and each mail out contained a sachet of coffee and a teabag as an unconditional, non-monetary incentive to participate. The surveys were administered between 3 February 2020 and 18 June 2020.

Table 1.  Examples of variations in wording of surveys (differences emphasised in bold).

The pre-specified primary outcome was the response rate, defined as the proportion of GPs in each group returning the completed survey by mail. A non-responder was defined as a participant who did not return completed study material after three postal reminders, or within 6 weeks of the sent date of the third reminder.

Assuming a maximum response rate of 27%¹¹ and a sample of 1875 GPs, we calculated that we would need a sample size of 434 per group to detect an 8% difference, with 80% power and 5% alpha.

The randomisation sequence was computer-generated with a 1:1 ratio and stratified by location (regional or metropolitan New South Wales) and practice (to mitigate the risk of contamination between study groups at practices). The random allocation sequence was assigned by a statistician who was not aware of the assignments, and the group assignments were allocated by the study coordinator.

The difference in response rates (primary outcome) between intervention groups was compared using the Chi-squared test on an intention-to-treat basis. The effect of other factors on response (sex and practice location) were analysed using logistic regression including interaction terms (sex-intervention group interaction and practice location-intervention group interaction). Raw and adjusted odds ratios were calculated, with 95% confidence intervals constructed around the estimates. Statistical significance was taken at P < 0.05 and analyses were conducted using SPSS for Windows Version 25 (SPSS Inc., Armonk, New York, USA).

Ethical approval for the study was sought and granted by the Sydney Local Health District Human Research Ethics Committee (SLHD HREC) – RPA Zone (Protocol Number X19-0366 & 2019/ETH00449). Site governance approval was granted by the RPA Hospital Governance Office (approval number 2019/STE16418).

The surveys were sent to 1875 GPs in total (930 in Group A ‘symptom-frame’ and 945 in Group B ‘cancer-frame’). Fig. 1 depicts the flow of the participants and responses over the study period. In total, 72 participants were excluded and did not receive the survey (n = 32 in Group A and n = 40 in Group B). The overall response rate to the survey was 10.9% (196/1803).

Fig. 1.  Flow diagram of participants through the study.

Response rates by group are presented in Table 2. In Group A, 100 participants completed and returned the questionnaire (11.1%), and in Group B, 96 participants completed and returned the questionnaire (10.6%). The difference was not statistically significant (0.5% [95%CI: −2.4, 3.4%]; P = 0.73).

Table 2.  Survey response rates by intervention group, gender, and location of practice.

When testing the effect of GP factors on response rates, practice location was the only factor that demonstrated an effect. GPs practicing in regional New South Wales were more likely to respond to the questionnaire (adjusted odds ratio [AOR] 1.61 [95%CI: 1.12, 2.31]; P = 0.01) (Table 3). However, there was no differential effect of the intervention by practice location. All P values for interaction terms were >0.05.

Table 3.  Odds ratio analysis of response rate – adjusted analysis.

We explored whether targeted wording within study materials sent to GPs impacted on response rates in a study of referral processes for head and neck cancer. We found no significant difference in response rates based on the wording of the questionnaire. However, a sub-analysis showed that response rates to the questionnaire were higher among GPs practicing in regional New South Wales compared to those practicing in metropolitan New South Wales, independent of the intervention group or participant sex (AOR 1.61 [95%CI: 1.12, 2.31]; P = 0.01).

Our initial hypothesis was that GPs who received the questionnaire using the ‘symptom-frame’ would be more likely to respond, as this may be more relevant in their practice than ‘suspected head and neck cancer’; however, we found no evidence to support this hypothesis. The interesting finding of a higher response rate among regional GPs after adjusting for participant factors warrants further consideration. The Leverage–Saliency Theory¹² proposes that if the content of a survey factors into the decision to participate, then the decision not to participate results in non-response bias. Groves, who proposed this theory, later demonstrated that response rates to a questionnaire with a specific topic (eg ‘education and schools’) vary based on the sampling frame.¹³ In this scenario, response rates to the survey were highest among the sampling frame that consisted of teachers. In contrast, a survey regarding ‘childcare and parents’ had the highest response rates from new parents. Although it is not possible to test this theory in the current study due to the absence of control groups, it is possible that the response rate was higher among regional GPs as they considered the general content (referral pathways for head and neck cancer) to be more relevant to their practice.

The primary limitation of this study is the low response rate (10.9%), and although other surveys have reported similarly low response rates among Australian GPs,¹⁴ this brings into question the generalisability of the results. However, a significant strength was that post hoc calculations showed that the study was still sufficiently powered, and we were able to detect a different in response rates based on location of practice. Another strength was the use of rigorous methods in the design of the questionnaire, which incorporated elements demonstrated to improve response rates to postal surveys.^5,6 It is possible that the response rate may have been even lower had these elements not been included in the design of the questionnaire and survey administration. However, recent evidence suggests that self-selection of survey mode (postal or electronic) is an effective method to maximise response rates in surveys of medical practitioners,¹⁵ which was not implemented in this study owing to limitations in obtaining emails of all eligible GPs. The survey was also administered during the peak of coronavirus disease 2019 (COVID-19) infections in New South Wales in early 2020, which may also have impacted on response rates to the survey.

The findings from this study highlight an ongoing challenge in surveying GPs concerning uncommon topics or conditions. Despite utilisation of evidence-based approaches to health-care surveys, the response rate was low (10.8%). This warrants additional investigation as to the optimal sampling strategies for surveys of GPs pertaining to uncommon topics or conditions, especially given the importance of high-quality studies needed to inform practice and policy. Additional insight into how best to engage GPs as research participants could lead to less research waste and improvements of translation of research into practice, resulting in mutual benefits for both research and practice.

To our knowledge, this is the first study to test differential wording on the response rate of GPs to a survey of cancer referral practices, and highlights the challenge of optimising response rates for surveys about uncommon topics or conditions. Although there is some evidence in the literature that questionnaire saliency can improve response rates to questionnaires, we did not find evidence of this in the current study. It is possible that the minor wording differences used in this study were too small to have any impact on response rates. Future studies testing this approach should account for this possibility in the design of questionnaires.

The data that support this study will be shared upon reasonable request to the corresponding author.

The authors declare no conflicts of interest.

This study received funding from the Primary Care Collaborative Cancer Clinical Trials Group (PC4) in 2018 under the PC4 Training Award. The funder played no role in the design of the study.