The Australian digital Online Farm Trials database increases the quality of systematic reviews and meta-analyses in grains crop research

Keywords: agriculture, Australia, crop research, cropping, database, FAIR principles, findable, grains, literature review, meta-analysis, metadata, OFT, sowing date, systematic review, time of sowing.

In the fields of academia and research, systematic reviews (sometimes referred to as systematic literature reviews) are generally accepted as a rigorous, formal, effective and repeatable type of literature review that can produce evidence-based conclusions. The strength of a systematic review lies in its rigorous, exhaustive and replicable method of reducing bias in identifying, selecting and evaluating all eligible studies that are drawn from numerous digital sources (Koutsos et al. 2019). When a systematic review includes the use of quantitative comparisons and statistical analysis of results, the review is referred to as a ‘meta-analysis’. A meta-analysis is considered to be the highest level of evidence that can be presented because it can be used to overcome problems associated with individual studies with reduced statistical evidence (Haidich 2010).

Originally developed for use within the field of medical research in the late 1970s, systematic reviews are being increasingly adopted across a wide range of other research disciplines, and the number of reviews and meta-analyses published is increasing exponentially (Haidich 2010). It is now considered necessary to conduct a systematic review before starting a new research inquiry as it identifies previous works and presents the current state of knowledge on the topic of interest. It will also guide the researcher to potential research questions where further work could provide novel insights.

Like many other research disciplines, the field of agricultural research contains a continually growing body of work (Cruz and Nascimento 2019). The increasing number of systematic reviews and meta-analyses published over the past ~15 years (Fig. 1) suggests that agricultural researchers have recognised these methods as an effective way of summarising past agricultural research, and that they will be adopted more widely in the future (Koutsos et al. 2019).

Fig. 1.  Number of publications with ‘agricultur*’ and ‘systematic review’ (solid line) or ‘meta-analysis (dashed line) in the title returned from the Web of Science database (accessed 2 September 2020).

Agriculture is one of the largest industries in many countries around the world with grain crops making up a large proportion of agriculturalproduction and trade (Ogundari and Bolarinwa 2018). The domestic and global demand for grain-based food and grain-fed meat is expected to increase substantially over the next 10 years (AEGIC 2019), putting agricultural research under increasing pressure to deliver meaningful results to drive increased crop yields within the context of sustainable development. In Australia, production of winter crops has increased since the late 1980s with the increase largely the result of expansion of areas sown to crops; however, further expansion of cropping areas between now and 2030 is expected to be modest (AEGIC 2019), so production increases will become increasingly reliant upon improving yields. Further, because most crops grown in Australia are rain-fed, changes in rainfall acutely affect crop production. But rainfall patterns (including the quantity and spatial and temporal distribution of precipitation) and shifts in air temperatures around the globe have been altered by anthropogenically-induced climate change, and predicted weather scenarios will continue to have substantial implications for agricultural production into the future (Kukal and Irmak 2018).

For crop productivity to be improved and sustained within a changing climate, Australian grain growers will need to adopt alternative management practices, new technologies and superior crop varieties developed to grow under various environmental scenarios. Each year, the Grains Research and Development Corporation (GRDC) receives around AUD200 million in investment and invests in hundreds of new research projects (https://grdc.com.au/research/areas-of-investment#:~:text = The%20GRDC%20receives%20around%20%24200,profitability%20of%20Australia’s%20 grain%20industry), the findings from which have enormous potential to improve both crops and management practices (Hyman et al. 2017).

Methodologies that are generally accepted for how to conduct a systematic literature review are based on The PRISMA Statement of Moher et al. (2009), with various adaptations proposed since its publication. At the most basic level, the general procedure includes searches of both academic (i.e. peer-reviewed and published ‘white’) and non-academic (i.e. ‘grey’) literature.

The academic literature base is searched using keywords and Boolean operators (e.g. ‘AND’, ‘OR’, ‘NOT’) in the search function of a digital database. There are many academic databases and search engines that can be used to search for published literature. Of the many databases available, there are two large, commercial and commonly used and internationally-recognised digital databases for academic literature that cover a broad range of scientific research fields: They are Scopus and Web of Science. Regardless of the exact database used, searches are based on the selection of terms (keywords or phrases) that can be combined with Boolean operators to produce the search query. Although this process sounds straight-forward, the use of appropriate keywords together with the selection of operators is of vital importance and will determine the results of the search (Koutsos et al. 2019). Incorrect or inappropriate selection may limit the number of eligible works returned by the search, which can lead to misleading results or unsubstantiated conclusions (Koutsos et al. 2019). Conversely, a poor search query may return so many articles that assessment is impossible or simply unfeasible, so careful planning and adjustment of search queries based on a pilot study review are needed to produce meaningful and appropriate search results.

However, relying solely on large digital databases of academic literature for a systematic review produces serious problems. First, there is generally a considerable time lag between when the research was conducted and the time it is published (Godin et al. 2015). The magnitude of the publishing lag depends on the research discipline, type of work and particular publication in question, but is usually in the order of months (rarely), and more commonly, to several or many years (Barbour 2020). Medical-based clinical trials have among the longest lag times (Riveros et al. 2013), whereas in the biological sciences (including agriculture), it may be a little as 6 weeks (The Open Agriculture Journal, https://openagriculturejournal.com/quick-track-option.php), although this is thought to be unusually short. Many publishers are attempting to reduce manuscript processing time (E. Dutton, pers. comm.), but a lag time of considerable length still generally remains. As a consequence, research trials may not be widely available until several years after they were produced, so conclusions may already be out of date at the time of review.

Second, academic works may suffer from what is known as ‘publication bias’ where peer-reviewed papers demonstrating a statistically ‘significant’ effect are more likely to be accepted for publication than those reporting negative or null results (Olson et al. 2002; Editorial 2019; Setter et al. 2020). This appears to be more common in some fields of research, with publications in the biological sciences falling between those in the physical sciences (which suffer a lower publication bias) and those in the social sciences (which suffer a higher publication bias; Fanelli 2012). Some newer journals are trying to overcome this problem by explicitly stating that they encourage publication of null results (e.g. f1000Research), but these journals are still the exception rather than the rule.

Third, many academic journal papers exist behind paywalls, requiring the reader to have a subscription to access (Barbour 2020). This limitation is slowly being reduced through increased adoption of the open access (OA) movement, which aims to ‘mak[e] all scholarly outputs freely available via the Internet, permitting any user to read, download, copy, distribute, print, search, or link to the full text of these articles, crawl them for indexing, pass them as data to software, or use them for any lawful purpose, without financial, legal or technical barriers other than those inseparable from gaining access to the internet itself’ (Open Access Australia, https://aoasg.org.au/what-is-open-access/). However, the implications of OA for researchers and scientific publishers remains an issue of great discussion (Open Access 2020). Authors are increasingly making use of pre-print servers (such as medRxiv and bioRxiv) where the first version of a manuscript can be deposited and made openly available within a day or two of submission.

These issues arise when using only academic literature for systematic literature reviews but they can be reduced by including work drawn from all available sources. Work that falls outside the realm of academic literature is generally referred to as ‘grey literature’ and can be defined in several ways. The most commonly accepted definition is that provided by the 1997 Luxembourg Convention on Grey Literature: [that] ‘is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publisher’ (Saleh et al. 2014). That is, where publishing is not the primary activity of the producing body (Godin et al. 2015) and where publication is not the primary aim (i.e. end goal) of the research undertaken. From these concepts, any unpublished research and/or data may be considered forms of grey literature (Higgins and Thomas 2019).

It is recognised that although grey literature is often neither peer-reviewed nor indexed in bibliographic databases (Tillett and Newbold 2006), it may still provide a source of good, reliable information (Enticott et al. 2018). For example, in intervention studies, exclusion of grey literature may artificially amplify estimates of treatment effects (Hopewell et al. 2007). Guidelines for conducting systematic reviews recently published by the Cochrane Collaboration (Cochrane Training 2021) stipulates that grey literature should be included in a systematic reviews. However, the quality and form of grey literature varies considerably, and it is generally included only if it meets the defined inclusion/exclusion criteria designed for white literature (McAuley et al. 2000), meaning that much of the grey literature is typically excluded.

To further complicate the adoption of its use, grey literature is difficult to search systematically (Godin et al. 2015). This is because it is: (1) not clearly defined or delineated; (2) not confined to a set or limited number of sources (and can therefore be vast and of unknown quantity); (3) difficult to identify possible relevant sources; (4) varies greatly within and between research disciplines; and (5) often available only as a digital resource (i.e. on the internet) without a persistent identifier (e.g. digital object identifier, doi). Most grey literature is accessed via digital means using internet search engines, which presents a real challenge for systematic reviews due to the huge (and continually growing) amount of published work, lack of standard indexing or controlled vocabularies (i.e. search metadata) and lack of archiving on the internet (Godin et al. 2015).

In contrast with searches of the white literature, there is no ‘gold standard’ to search for grey literature (Godin et al. 2015). Further, reporting of grey literature search methods are not generally held to the same high standards of transparency and reproducibility as white literature search methods (Briscoe 2015). However, it is critical that the searches used to identify studies for inclusion in a systematic review should be comprehensively reported to ensure they are transparent and reproducible (Briscoe 2018). In many instances, the Cochrane Handbook for Systematic Reviews of Interventions (https://training.cochrane.org/handbook) is cited as the best resource for how to conduct systematic reviews, and it includes limited guidance for grey literature search methods. But as the contribution of supplementary search methods in systematic reviews is increasingly acknowledged (Cooper et al. 2017), a growing number of authors are reporting methods that have been implemented to successfully identify relevant grey literature within various disciplinary fields (e.g. Godin et al. (2015), Garousi et al. (2019), Enticott et al. (2018), Cooper et al. (2018)). The development of more specific guidelines for academics and researchers on how to include grey literature searches in reviews will be important as the practice of systematic review in the field continues to mature (Garousi et al. 2019). Research investigating the effectiveness of search methods for health-related grey literature have been well reported, and social sciences have also been recently considered (Stapleton et al. 2020), but methods for conducting systematic searching of grey literature within agricultural research have not been reported. Indeed, the framework proposed for conducting systematic literature reviews in agricultural sciences by Koutsos et al. (2019) did not include grey literature searches.

Within the field of agricultural research, a large amount of experimental trial work is field-based, and is conducted by non-academics, so most works are neither submitted nor published in academic journals (K. Light, pers. obs.; N. Scholz, pers. comm.) and findings are often unavailable to those not directly involved in the research (Murphy et al. 2015). Searches of the grey literature are thus key in agriculture-related systematic reviews and meta-analyses. Online Farm Trials (OFT) is an open digital database containing information and results from grain crop research trials of over 15000 ‘crop-type × location (site) × year combinations’ (called ‘Trial Projects’) across Australia. It was developed in late 2013 to address the need for greater dissemination and accessibility of on-farm research trials (Walters et al. 2018). OFT is funded by the GRDC but includes research trials conducted by a large and diverse range of contributors including farming and grower groups (e.g. Birchip Cropping Group, Southern Farming Systems, Mallee Sustainable Farming, Northern Grower Alliance), state government departments (e.g. Department of Primary Industries and Regional Development, Department of Agriculture and Food WA, NSW Department of Primary Industries), research organisations, programs and alliances (e.g. South Australian Research and Development Institute, Eyre Peninsula Agricultural Research Foundation, Southern Pulse Agronomy, Soils West), agricultural contractors and service providers (e.g. Agrarain Management, Planfarm, agVivo) and universities (e.g. The University of Adelaide, Federation University). Research included is largely independent and any group or individual wishing to enter information may do so free of charge. The database was developed and is maintained by Federation University Australia. Thus, there is potential for OFT to serve as a discipline-specific database of agricultural grey literature. Discipline-specific databases such as clinical trial registers have shown to be useful for informing systematic reviews in other disciplines (Briscoe et al. 2020), and thus address some of the key limitations of systematic reviews based on white literature alone.

In this work, we used the discipline-specific OFT database and the multidisciplinary Web of Science database to identify information for a systematic literature review and/or meta-analysis based on a hypothetical research question. We wanted the focus question to have a real-world application, formulated from the observation that agricultural production is being increasingly impacted by the effects of anthropogenically-driven climate change, and methods for increasing yield gains are needed if projected demands for grain are to be met. For grain growers, time of sowing (i.e. sowing date) is one of the key management methods farmers can use to influence crop yield in rain-fed winter cropping zones: optimum timing is a delicate balance between sowing too early (with an increased risk of frost damage) and sowing too late (with an increased risk of drought and heat damage). Australian crops are being sown earlier than in the past (Stephens and Lyons 1998; GRDC 2011; Fletcher et al. 2016; Walters et al. 2020) but we could find no assessment of whether earlier sowing has led to increased grain yields. Here, we used the question ‘has earlier sowing of crops in Australia research trials led to increased grain yields?’ in a case study approach leading an investigation into the value of using the OFT database in identifying information for a systematic review and/or meta-analysis. The objective of this study was to investigate the amount of information available on this topic and the potential contribution of grey literature sourced from OFT to informing systematic reviews and meta-analysis in Australian grains research.

All information was gathered online between 1 July 2020 and 30 November 2020. The review protocol used was based on the framework for systematic reviews in agriculture as presented by Koutsos et al. (2019), which extends the general PRISMA statement (Moher et al. 2009) and includes the following steps: (1) scoping; (2) planning; (3) identification; (4) screening; (5) eligibility/assessment; and (6) presentation (synopsis of findings, discussion and presentation of results). The final step (6) was not completed in its entirety because we did not conduct an actual review or analysis of the identified literature or data.

The three main parts of scoping are: (1) statement of the ‘focussed question’; (2) a search for previous systematic reviews on the identified issue; and (3) identification and use of a few relevant studies for a pilot review study. Here, we asked the hypothetical focussed question: to what extent does earlier sowing date (time of sowing) increase grain yield in Australian crops? We searched for previous systematic reviews and meta-analyses of sowing date and grain yield in three large, comprehensive, multidisciplinary digital databases: (1) Web of Science (using the search terms: [‘systematic review’ OR ‘meta*analysis’] and [‘time of sowing’ OR ‘sowing date OR ‘sowing timing’] in the title); (2) Scopus (using the search terms: [‘systematic review’ OR ‘meta*analysis’ AND ‘sowing timing’ OR ‘time of sowing’ OR ‘sowing date’] in the title); and (3) Google Scholar (using the search terms: [time of sowing systematic review], [time of sowing meta-analysis], [sowing date systematic review], [sowing date meta-analysis], [sowing date systematic review], and [sowing date meta-analysis]. Searching on titles was chosen as the most effective way of identifying papers for consideration as searches on topic returned many works of no relevance. (Note: the truncation ‘meta*analysis’ was used as the root term to retrieve variations on ‘meta-analysis’ and ‘meta analysis in databases where * is a functional truncation symbol.)

The Google Scholar search provided one review titled ‘Nordic agriculture under climate change: a systematic review of challenges, opportunities and adaptation strategies for crop production’ by Wiréhn (2018). This work was assessed for relevance but it does not attempt to answer the focus question proposed for the present study so was not investigated further.

We then conducted a scoping search using the same three databases but rejected investigation of other commonly used bibliographic databases (e.g. Medline) because they were deemed irrelevant to agricultural research. From the scoping results, the Web of Science database provided the best source for published work on the focus question. This observation agrees with previous searches for work within the agricultural sciences (Koutsos et al. 2019) and makes use of findings by Gusenbauer and Haddaway (2020) who determined that the Web of Science database is one of the academic databases that is well suited to evidence synthesis in the form of systematic reviews.

In the planning stage, the key steps employed were to: (1) identify appropriate digital databases or sources of eligible information; and (2) develop the search strategy and build the search queries.

The Web of Science digital database was selected as the source of eligible peer-reviewed, published (i.e. ‘white literature’) studies The OFT digital database was used as the discipline-specific database to search for eligible grey literature.

The aim of this study was to identify works that have investigated the effects of sowing timing on yield of the grain crop species in Australia. We focussed on Australian winter crop species; namely barley (Hordeum vulgare L.), canola (Brassica napus L.), chickpeas (Cicer arietinum L.), faba beans (Vicia faba L.), field peas and kaspa peas (Pisum sativum L.), lentils (Lens culinaris L.), linseed (Linum usitatissimum L.), lucerne (Medicago sativa L.), lupins (Lupinus spp.), mustard (Brassica spp.), oats (Avena sativa L.), triticale (× Triticosecale. Wittm. ex A. Camus.), vetch (Vicia spp.) and wheat (Triticum aestivum L.) because sowing dates of these species have been moving earlier within some Australian states over the past ~40 years (Stephens and Lyons 1998; Fletcher et al. 2016; Walters et al. 2020), allowing for investigation of the hypothetical focussed question stated in step (1).

For a work to be eligible for inclusion in a systematic review (i.e. review), it had to: (1) have the purpose of examining the effects of different sowing dates on crop grain yield, or comment on sowing timing vs grain yield; (2) pertain to one of the specified crop types under investigation (see above); (3) be a rain-fed crop; (4) reporting research conducted within Australia; (5) be published in English; and (6) be available to the authors online. Studies that did not report the effects of sowing date/time of sowing as treatment that could be separated from other treatments were excluded, as well as studies that were based on modelling, simulation or forecasting. For a work to be included in a meta-analysis (i.e. analysis), it had to contain numeric data in tabulated format.

For the identification search, we used the Web of Science to locate records linking time of sowing and grain yield in Australia using the search terms: [‘management’ OR ‘time of sowing’ OR ‘sowing date’ OR ‘sowing timing’] AND [‘sow*’] AND [‘yield’] AND [‘Australia’] in the topic. This search returned a total of 117 papers, one of which was a duplicate and was removed, the rest were used in the screening step (see below).

The search method for the OFT database uses metadata elements as filters for available Trial Projects. Filtering on metadata elements is a binary decision-making process (i.e. a filter is either selected or not selected) and does not require the selection of keywords or Boolean operators.

The ‘Advanced search’ function of OFT (https://www.farmtrials.com.au/Advanced-Search, accessed 2 September 2020) was used to filter Trial Projects according to the following metadata elements: Trial content = ‘Trial report available’ and ‘Without adverse events’; Treatment type = ‘Sowing timing’ Crop type = ‘barley’, ‘canola’, ‘chickpeas’, ‘faba beans’, ‘field peas’, ‘kaspa peas’, ‘lentils’, ‘linseed’, ‘lucerne’, ‘lupins’, ‘mustard’, ‘oats’, ‘triticale’, ‘vetch’, and ‘wheat’. Results from the OFT database search returned a total of 263 Trial Projects for inclusion in the screening step (see below), and a list of Trial Projects was downloaded to an MS Excel spreadsheet via the export function in OFT.

The screening of returns obtained from the refined search of Web of Science was undertaken independently by both the authors following the eligibility criteria outlined above. To decide whether a paper was eligible for inclusion in a review, we first examined the title of the paper. If the information in the title was insufficient for the basis of a decision, then the paper was retrieved. A total of 52 papers were marked for retrieval, one of which could not be found. For the papers retrieved, the abstract was reviewed and if it did not provide sufficient clarification to determine inclusion or exclusion, the main text of the paper was then consulted. For papers eligible for review, we then assessed whether it was eligible for inclusion in a meta-analysis. For this, it needed to contain numerical data relating to sowing date vs grain yield in tabular format.

After independent screening, results from both authors were compared and where discrepancies occurred, the authors re-assessed the work together to reach a unanimous agreement. From the 116 papers screened, 20 (~17% of the number screened) were eligible for inclusion in a review, and of those, nine were eligible for inclusion in an analysis. Finally, from the papers eligible for meta-analysis we investigated those that reported a ‘null’ result. Papers were reviewed and classified as either reporting a null result or reporting statistically significant differences (i.e. not a null result). A list of the 20 papers eligible for review and/or analysis is provided in Supplementary material Table S1, available at the journal’s website.

The list of Trial Projects eligible for screening included the metadata elements of ‘Trial name’, ‘Year’, ‘Crop type’, ‘Location’, ‘Organisation’, ‘Aims’ and ‘Key messages’. Examination of the accuracy of information contained in the ‘Sowing timing’ metadata element of the 263 Trial Projects selected showed that one was a duplicate and 16 did not include information comparing different sowing dates (i.e. they had been incorrectly tagged as sowing timing). Hence, these were excluded before screening. Also, there were two Trial Reports providing the same information (having been entered into the database by different contributors), and in this case, only one Trial Report was included in the analysis.

The remaining 246 Trial Projects were then assessed for eligibility to according to the same criteria as for the Web of Science papers (see above). Briefly, Trial Projects were assessed by first checking the title, and if that did not provide a definitive answer, the key messages were assessed. If this failed to provide enough information, the trial report provided as a PDF was accessed via the OFT website (https://www.farmtrials.com.au). For a trial to be considered a ‘sowing timing’ trial, there had to be two or more sowing dates included as part of the research design. From the 246 Trial Projects screened, 239 (~97% of those screened) were eligible for inclusion in a review. A list of the 239 Trial Projects eligible to be included in a systematic review is provided in Table S2.

Eligibility for inclusion in a meta-analysis was then determined by checking: (1) individual trial reports and including those that contained numerical yield data testing the effects of sowing date (i.e. the results had to compare yield at the first time of sowing (TOS1) with yield at the second time of sowing (TOS2)); and (2) identifying Trial Projects that had numerical results data available for direct export. This was done by using the results from the search above and checking the filter for Trial content = ‘Export data available’. In total, 147 (~60% of those screened) Trial Projects were eligible for inclusion in an analysis. Of these, 69 had relevant data (i.e. sowing date vs grain yield) available for direct export (see Table S2).

We next reviewed and classified trials as either reporting a null result or reporting statistically significant differences (i.e. not a null result) from the Trial Projects eligible for analysis. (Note: the direction of the difference was irrelevant for this part of the investigation; i.e. we were not testing the hypothesis that earlier sowing resulted in increased grain yield). Some Trial Reports that reported both null and statistically significant results (e.g. some varieties produced higher yield with earlier sowing, but others did not), so these were recorded as reporting both a null result and a statistically significant result.

Finally, we investigated the statistical rigour by applying the ‘Trial design’ filter in the OFT Advanced Search function from the Trial Projects eligible for analysis. Trials can be entered as ‘Blocked’ and/or ‘Randomised’ and/or ‘Replicated’, the most rigorous designs being ‘Blocked, randomised and replicated’. We counted the number of Trial Projects that were Blocked, Randomised and Replicated, as well as those that were Replicated only. Results are shown in Table S2.

The study flow of search results from each phase of the systematic review is in Fig. 2. The total number of records available from OFT was far greater than from the Web of Science, so comparison of these numbers and percentages should be treated carefully even though the process of determining inclusion and exclusion was the same for both databases. The number of Trial Reports identified from OFT far exceeded the number of published papers from the Web of Science: 263 vs 117. Each of the databases had one duplicate to be removed. The percentage of OFT records screened that were eligible for inclusion in a review was greater than the percentage from the Web of Science (97 vs 17%), and a similar difference was seen for records eligible for inclusion in a meta-analysis (60 vs 8%).

Fig. 2.  Study flow diagram.

The number of eligible records (i.e. papers from Web of Science and Trial Projects from OFT) investigating the various winter crop types is in Table 1. There were more records for wheat than any other crop type in both databases: OFT returned almost eight times as many records than Web of Science (85 vs 11, respectively). The only other crop type that Web of Science returned more than a single record was canola, for which three records were identified. OFT identified 10 times as many trials (i.e. 30) for canola. The OFT database also returned more than 20 records for barley, chickpeas, faba beans, field peas and lentils, whereas Web of Science returned only single records for all these species.

Table 1.  Number of papers (from Web of Science) and Trial Reports (from Online Farm Trials) eligible for systematic review for each crop type.

The publication years of Web of Science papers ranged from 1993 to 2020, and study years from 1987 to 2013 (see Table S1). Papers had a 4.5 year average lag time from the end of the study to publication date, with lag times varying between 2 years (minimum) and 11 years (maximum). The experimental work conducted for the most recently published paper (Manning et al. 2020) was completed in 2013, so the newest information available via the Web of Science was 7 years old at the time of publication.

Trial Projects in OFT are filtered on growing season year rather than publication year. For example, a Trial Project entered as 2018 will have been conducted in 2018. The trial report booklets from which OFT Trial Projects are generally sourced are published within 12 months of completion of experimental trial work (J. Walters, pers. obs.) and can be entered into the OFT database at any time after publication.

There was an obvious difference in the number of records eligible for systematic review and /or meta-analysis from the two databases that reported a null result. Table 2 shows the number of eligible papers from Web of Science and Trial Reports from OFT together with those reporting ‘null, ‘null and significant’ or ‘significant’ results. Of the 19 papers eligible for review from the Web of Science, one reported a null result (i.e. there was no difference in yield with time of sowing) whereas the other 19 papers (95%) reported a statistically significant difference (i.e. time of sowing produced a statistically significantly different grain yield). For OFT, 57 of the 239 Trial Projects (24%) reviewed reported a null result and a further 24 (10%) reported both ‘null’ and ‘significant’ results. Most of the Trial Projects that reported both null and significant results were comparisons of crop varieties, where the yields of some varieties were affected by sowing date but yields of other varieties were not. A similar pattern was detected for meta-analysis records: Only one of nine papers (11%) from Web of Science reporting a null result compared with 36 of 147 Trial Projects (25%) from OFT.

Table 2.  Number of records (%) from Web of Science and Online Farm Trials eligible for inclusion in a systematic review (Review) and meta-analysis (Analysis) that reported a ‘null’ result and/or statistically ‘significant’ result.

Assessment of the level of statistical rigour in OFT Trial Projects using the ‘Trial design’ filter showed that of the 239 Trial Projects eligible for review, 112 (~47%) were of the highest possible level of experimental rigour, being entered as ‘Blocked, randomised and replicated’, and a further 68 (~29%) were of the second highest level of rigour, being entered as ‘Replicated’. In total, ~75% (180 of 239) of Trial Projects were considered to report results with high statistical rigour.

Systematic searching of the OFT database identified more than twice as many records for investigation as the Web of Science database (263 vs 117, respectively), indicating that there is a large amount of information available via the OFT database. For Web of Science, 96 of the 116 records reviewed (after removal of one duplicate) were rejected. This was a higher rejection rate than for OFT, for which 23 of the 262 records reviewed (after removal of one duplicate) were rejected. Thus, the percentage of records eligible for inclusion in a systematic review was much higher for OFT than Web of Science: 90% of records identified via OFT were eligible for inclusion versus only 18% of records from Web of Science. This shows a benefit of using the OFT database to locate records for screening, which is further highlighted when considering the number of records identified for each of the major crop types (Table 2). For the three most economically important Australian crop types (wheat, barley and canola, based on acreage sown and tonnage produced) included in the study, the OFT database returned 85, 29 and 30 Trial Projects, respectively, compared with 11, one and three studies from the Web of Science database. The subset of records eligible for inclusion in a meta-analysis showed a similar pattern, with 55% of OFT records eligible for inclusion in a meta-analysis compared with only 8% from Web of Science. These observations demonstrate that the amount of relevant information available for use in both systematic literature reviews and meta-analyses of cropping research in Australia would be bolstered greatly through the inclusion of grey literature discovered using OFT. This confirms the previously suggested importance of including grey literature in several fields of research such as public health (Adams et al. 2016), as well as in future reviews and analysis of crop research in Australia.

The relative contributions of white and grey literature to the outcome of a review or analysis can alter the outcome of an investigation (e.g. Godin et al. 2015). Because we did not use the search to conduct a review or analysis here, further investigations are needed to determine whether the finding from a systematic review or meta-analysis could be altered by inclusion of records located via OFT. It is likely the answer may depend strongly on the focus question being investigated and the relative difference between the number of studies included from each source.

We observed that the time required to conduct searches of the two databases was markedly different. Use of filtering on metadata elements in OFT searches was quicker than employing keywords and Boolean operators in Web of Science. The time required for the OFT search process was ~3.5 h, which equates to around 52 s per eligible record for a systematic review or ~1.4 min per eligible record for a meta-analysis. The Web of Science search required ~5.5 h, which equates to around 15.7 min per eligible record for a systematic review or 36.7 min per eligible record for a meta-analysis. This difference represents a better return of investment for time when using OFT compared with the Web of Science. A similar observation has been made by another researcher using OFT to gather information and data from trial reports (A. Schapel, pers. comm.; see https://www.farmtrials.com.au/videos/). Other sources of potential agricultural grey literature include grower group websites, state government department website and repositories, and university libraries. Searches of grey literature are noted as being more time consuming than database searches for white literature (Stapleton et al. 2020) and there are currently no recognised methods for searching these sources within the agricultural sector. Further, references located via OFT can be cited in journal publications according to the Terms of Use (https://www.farmtrials.com.au/terms-of-use/). OFT can be referenced as a general source of information or researchers may cite individual trial projects or data exports according to general publication standards. Thus, using grey literature sourced via OFT provides a notable advantage over similar literature found through alternative search methods.

Further, OFT searches are subject to binary decision-making processes (i.e. a metadata element filter is either selected or not selected). This mode of search differs from the more traditional search of a bibliographic platform such as the ScienceDirect (http://sciencedirect.com, accessed 14 August 2020) or ISI Web of Knowledge (www.webofknowledge.com, accessed 14 August 2020) that are most commonly used as the basis of a systematic review, where works were included if they contain the chosen keyword(s) or phrases contained within the chosen metadata fields (e.g. Title, Abstract, Keywords; Beckmann and von Wehrden 2012). The OFT filtering process also differs from the search of a platform such as Google Scholar (http://scholar.google.com, accessed 14 August 2020) where returns are not limited to metadata, but are drawn from every text that is electronically available on the internet (Beckmann and von Wehrden 2012), assuming that it is indexed (Falagas et al. 2008). Although these searches are comprehensive, Google Scholar can return a vast corpus of available literature, which makes the review process highly challenging (Gibert et al. 2016) and often infeasible.

The use of metadata elements as filters for a database removes the need to determine the most appropriate keywords or phrases and operators, thus eliminating problems associated with poor search queries as highlighted by Koutsos et al. (2019). We showed that the filters in OFT provide a simple and effective way of selecting relevant works; however, we note there are some potential disadvantages of this search method. First, the accuracy of returns is reliant upon accurate and appropriate entry of metadata in the database. In this work, 17 of the 263 Trial Projects (~6%) returned in the OFT search were excluded before screening because their associated Trial Reports did not include information comparing different sowing dates; i.e. they had been incorrectly tagged as sowing timing. Overall however, even if this number is added to the further seven Trial Projects rejected, it remains a small percentage of returns rejected (~9%) compared with the larger percentage of returns rejected from Web of Science (~83%). Thus, this suggests future searches simply include a check for correct tagging after export of search results, which will remove any possible errors introduced by incorrect entry of metadata elements. Second, filtering on metadata elements introduces a lack of flexibility in the search results that does not exist when using keywords and operators. However, returns can be refined by filtering on further metadata fields including ‘Crop type’, ‘Year’, ‘Contributor’, ‘GRDC Region’, ‘Trial Site’ and ‘State’. Application of further filters would enable returns to be refined and search results focussed to meet user requirements.

This investigation demonstrates that using the OFT database also helps to overcome several problems identified with methods that rely on traditional searches of the academic literature using traditional multi-discipline databases such as the Web of Science. First, there is the capacity for a reduction in publication lag time. OFT Trial Projects can be uploaded by a contributor as soon as the grower group or researcher has completed and finalised the trial report. Trail Report books are usually produced annually, so results from one year’s research is available within 12 months of completion. In contrast, published work available in the academic literature accessed through the Web of Science had an average lag time of 4.5 years from the last year reported in the study to the year of publication. The shortest lag time was 2 years and the longest was 11 years. By comparison, our OFT search identified 109 Trial Reports for research that has been completed since 2013. Thus, OFT provided a far greater resource of up-to-date research results than the academic literature.

Second, there was a higher proportion of records eligible for review and analysis from OFT that reported a null result than for the Web of Science, thus helping to overcome the problem of publication bias (e.g. Setter et al. 2020). Filtering out work that has been identified as impacted by ‘adverse events’ increased the reliability of results from the OFT database. Adverse events are defined individually for each Trial Project (where they exist), and include severe frost, insect or herbivore damage or extreme heat events that impact the trial results. Of the 239 Trial Projects from OFT eligible for systematic review, 69 reported a null result and an additional 57 reported mixed (i.e. both null and significant) results from different crop varieties: 112 Trial Reports reported statistically significant results; however, this was a smaller majority than for papers from the Web of Science, where only one of the 20 eligible papers reported a null result. None of the papers from the Web of Science reported mixed results. For meta-analysis records, 36 of the 147 Trial Projects from OFT reported a null result, and a further 20 reported mixed results: 91 reported a ‘significant’ effect: from the Web of Science, eight of the nine meta-analysis papers reported a statistically significant effect of sowing date and only one reported a null result. This shows that records eligible for inclusion in a meta-analysis from OFT are at least twice as likely to report a null result than those in the academic literature. Inclusion of Trial Reports from OFT in a meta-analysis would help alleviate a potential publication bias in favour of ‘significant’ results that can exist when relying on published studies in the academic literature (Haidich 2010).

Third, Trial Reports available via OFT are free to access with no login requirement, subscription fee or paywall. The information is available to anyone with an internet connection and can be read, used and reused anywhere around the world. The majority of Trial Projects are licenced under a Creative Commons Attribution 4.0 International Licence (CC BY 4.0), so information can be used and reused with appropriate attribution. Other projects are protected by copyright but can be used and reused for usual practice under copyright law. Most of the records identified via the Web of Science were from published journals, which can be accessed only by researchers or others with the relevant login or subscription, usually provided by a university or employer. This means availability is limited, and those without full access ability would have an incomplete search, thus affecting any conclusions drawn from the records retrieved. This limitation would apply to both systematic reviews and to subsequent meta-analyses.

For meta-analyses, OFT provides the option of direct export of numerical data for further analysis. A total of 69 of the 147 Trial Projects (~47%) eligible for inclusion in a meta-analysis had data for yield vs sowing date available for export. This feature represents a considerable time saving for further analyses because this data would usually have to be entered manually into a spreadsheet for further analysis as is usually the case for a meta-analysis from published papers.

Several problems have been identified in the processes previously used for searching grey literature. Tillett and Newbold (2006) noted that grey literature is not usually indexed in a database, so OFT is novel in its very provision of grey literature in a searchable digital database. Godin et al. (2015) identified several additional issues noting that grey literature is: (1) not clearly defined; (2) not confined to a set number of sources; (3) difficult to identity within sources; (4) varies greatly in quality even between sources; and (5) is in digital form but is often transient nature and lacks a permanent identifier. The OFT database negates several these difficulties: (1) the website and individual Trial Projects are clearly defined); (2) contains information in a single (i.e. is a ‘one-stop-shop’ for cropping research) source; (3) needs no further identification within the source; (4) indicates quality of research with filterable metadata elements (see below for further information); and (5) the OFT website has a permanent identifier (doi:10.25955/5d4a53ed4254c). Thus, OFT enables reproducible search results for cropping-based grey literature in the same way as the databases such as the Web of Science provide reproducible search results for academic literature, so provides a method for systematically reviewing the Australian crop research grey literature.

The use of grey literature in systematic reviews and meta-analyses is often rejected because of a lack of reliability of results or sufficient statistical rigour (Haidich 2010). Our investigation into the statistical rigour of OFT Trial Projects showed that 114 of the 239 (~50%) of Trial Projects eligible for review were of the highest possible level of statistical design rigour (indicated as being ‘Blocked, randomised and replicated’) and thus should be comparable with the standard usually required of peer-reviewed scientific papers. These could be considered as ‘1st tier’ grey literature in the ‘shades of grey’ literatures suggested by Adams et al. (2016). However, Trial Reports from these Trial Projects may still not be suitable for equal weighting with information found via an academic databases due to a lack of peer-review in most cases. A further 78 (~33%) were entered as ‘Replicated’, indicating that there was at least some degree of statistical rigour (i.e. replication of plots and subsequent calculation of variation) in the trial design and subsequent analysis of results. These Trial Reports could be considered as ‘2nd tier’ grey literature (Adams et al. 2016). These findings show that most of the results from OFT Trial Projects could be confidently used in further analyses, although individual datasets should still be carefully checked and interpreted appropriately.

Among the many benefits of using OFT outlined above, we acknowledge several limitations and potential disadvantages in use of the database. First, Trial Reports are limited to Australian cropping research trials as no international works have yet been included. Thus, use of OFT is likely to be of greatest relevance to Australian crops and use in global reviews or analyses should note this geographical constraint. Second, OFT data is provided by participating contributors and users are alerted to the fact that information available via OFT may not be accurate, current or complete. Information is subject to change without notice, frequently being validated, enhanced and updated, and is subject to the usual uncertainties of scientific research. However, there are several quality assurance and quality control measures used to maximise quality of the data and the accessibility and usability of the database (Wills et al. 2018). For example, each Trial Project must contain information in a minimum set of metadata fields, namely ‘Trial project code’, ‘Trial project title’, ‘Growing season year(s)’, ‘Trial site(s)’, ‘Crop type(s)’, ‘Trial type’ and ‘Treatment type(s)’. Additional information such as the trial aim, key findings and trial report (which can be attached as a PDF) are strongly encouraged. Further, all data entered must be approved for publication by the contributing organisation and notifications of trials published alert the database team that a new trial has been published. However, when using OFT as a source of information, each Trial Report should still be assessed to determine whether reporting standards are adequate for the purpose intended.

In this work, no attempt was made to gather grey literature from sources other than OFT because there are several published methodologies that provide guidance on the use of supplementary search methods. Five core supplementary search methods used by the most influential handbooks in informing systematic review practice in the UK have been identified: (1) trial register searching; (2) hand-searching; (3) web searching; (4) contacting study authors or experts; and (5) citation chasing. The applicability and effectiveness of each of these methods is discussed in detail by Cooper et al. (2017) and researchers embarking on the review process are advised to consult the literature to determine the most appropriate methodology for their particular area of interest. Further, we suggest that use of the OFT database fulfils the core methods (1), (2) and (3), and serves as a precursor to (4). Specifically, OFT is a database of Australian crop research trials, thus searching OFT replaces the need to conduct hand searching of individual trial report books or extensive web searching of grower groups, researchers, universities and government agriculture research departments because trials from all of these sources exist in OFT. The results from an OFT search can also be used to follow up specific contributors or researchers (4) for further information, and contact details for each of the 64 contributors with published Trial Projects are available on the OFT website. Finally, citation chasing (5) is more usually applied to white literature, where use of the ‘cited by’ function in a database such as Scopus may provide further papers for screening. But for OFT, searches filtered on ‘Year’ can provide a record of research conducted during any specified time period, so neither forwards nor backwards citation chasing is necessary to obtain the most up-to-date research. Alternatively, the ‘Related program’ filter can be used to locate other Trial Projects that may be of relevance.

The increasing number of systematic literature reviews and data meta-analyses within the agricultural sciences has the potential to provide much needed synthesis of research data to assess past results, inform future studies and develop farm management guidelines and adoption of best practice activities to increase grain crop yield in Australia. Outcomes of these works can be compared with the use of reviews and analyses in a health-care setting where they are often read by clinicians to keep up-to-date in their field and are often used as the starting point for developing clinical practice guidelines (Moher et al. 2009). The traditionally-recognised methodologies employed to conduct systematic literature reviews and meta-analyses are well suited to areas of research where most studies are published in peer-reviewed journals, and where a large number of works are published each year. In these fields, published papers (i.e. white literature) form the bulk of the total literature available on a topic. However, our investigation has confirmed that although there is a large number of crop research trials conducted each year, the number of studies published in the academic literature is small. Thus, a considerable portion of research output is neither submitted to nor available via the traditional publishing pathway. This finding is consistent with previous observations that crop research studies are more usually reported in research trial report books produced by grain-grower or research groups. Many of the trial reports have been made available via OFT and this work has shown that there is considerable value in the research findings from unpublished trials to inform systematic literature reviews and data meta-analyses within Australian grains research. We suggest that future reviews and analyses of grain crop research use OFT as a source of valuable grey literature to provide a more comprehensive picture of recent and historical findings to help drive improvements in grain crop production.

JRW is employed as a freelance editor by CSIRO Publishing but did not have any part in the review, acceptance or publication of this paper. Crop & Pasture Science encourages its editors to publish in the journal and they are kept totally separate from the decision-making process for their manuscripts. The authors have no further conflicts of interest to declare.

This work did not receive any specific funding.