The value of research: using the Impact Tool to evaluate realised and anticipated benefits of the Cooperative Research Centre for Beef Genetic Technologies

Keywords: Beef CRC, impact assessment, Impact Tool, return on investment.

The Cooperative Research Centre (CRC) for Beef Genetic Technologies operated for its third successive 7-year term from July 2005 to June 2012. It aimed to achieve an additional 1.5% p.a. in gross revenue of the Australian beef industry over 25 years to 2030 by developing new genetic and genomic technologies and non-genetic ‘products’ (practices, processes, tools and technologies) based on a combination of genetics and cattle management to improve profitability, productivity, animal welfare and responsible resource use of Australian beef businesses.

The economic analyses underpinning the 2004 business case for the CRC’s third term (CRC for Cattle and Beef Quality 2004) were published in Griffith et al. (2006) and Griffith (2009a). Those analyses were based on a broad ‘top-down’ modelling approach, supported by detailed ‘bottom-up’ analyses of specific project areas. Subsequently, the Commonwealth Government requirements for monitoring and evaluation of CRCs became more rigorous and prescriptive. An Excel-based software package known as the Impact Tool (CRC 2013) was developed and mandated for use by all CRCs when submitting bids for funding, reporting annual progress against milestones and final reporting and evaluation of achievements against objectives. The Impact Tool is simply a template that collates and aggregates a range of data across all output and outcome areas specified by the CRC. As such it relies on very detailed bottom-up analyses of all of these areas.

The purpose of this and a related paper (Griffith et al. 2013) is to determine how well the third-term Beef CRC met its objectives at the end of its funding period. As a rigorous top-down analysis had been undertaken for the 2004 business case, and the Beef CRC had been routinely reporting to the Commonwealth using the Impact Tool since 2010, a three-pronged approach was adopted by the Beef CRC to evaluate its impact (CRC for Beef Genetic Technologies 2012).

First, several economic analyses were conducted of the potential impacts of most of the specific outputs that the CRC had targeted. These were new ‘products’ such as estimated breeding values (EBVs) to identify genetically superior cattle for breeding, DNA-based diagnostic tests, decision support tools and information packages, etc. The analyses were detailed investigations of how the individual products would be used in beef farming systems or in beef value chains. They could be termed ‘on-the-ground’ analyses.

Second, the Beef CRC used these product analyses as inputs into the Impact Tool, where they were aggregated and used to generate return on investment values. The Commonwealth Government requires the Impact Tool to be used as an overview of the outputs and outcomes from the CRC at the program level, and asks analysts to take a ‘helicopter’ view.

Third, after the winding up of the Beef CRC, the 2004 business case was repeated using exactly the same model and simulation processes, but substituting parameter values based on 2012 information from the Impact Tool and the detailed studies underpinning it, replacing the 2004 estimates. By its nature, it is an industry-wide view of the outcome of the whole CRC, and could be termed an ‘airliner’ view.

The second level of evaluation is reported in this paper. The focus is on how the new information generated by the CRC’s R&D activities between 2005 and 2012 has impacted on the predicted overall outcome of the CRC, as measured by application of the Impact Tool. This analysis is then used as an input into the third level of evaluation, reported in Griffith et al. (2013). The paper concludes with some lessons learnt from working with the Impact Tool.

The CRC for Beef Genetic Technologies was a collaborative effort between 21 partner institutions from Australia, United States, Canada, New Zealand, Korea, Brazil, Ireland, France and South Africa. It commenced in 2005 and focussed on development of new genetic and genomic technologies to address beef industry priority issues to improve profitability, productivity, animal welfare and responsible resource use of Australian beef businesses (CRC for Cattle and Beef Quality 2004).

The Beef CRC employed emerging genetic technologies based on DNA sequence information to develop new tools and technologies to enable the Australian beef industry to:

• Improve the capacity to deliver high quality beef to Australia’s 110 global markets using cattle of known genetic merit for exacting specifications, without compromising animal welfare or the environment;

• Enhance beef yield and herd reproductive efficiency, improve efficiency of resource use, reduce production costs, minimise methane emissions and avoid chemical and antibiotic residues through precise application of knowledge about the genes controlling these attributes in cattle, their rumen microorganisms and in parasites that affect cattle productivity; and

• Ensure Australia is the number one supplier of beef to meet the growing demand by neighbouring Asian countries to 2020.

Operationally, the Beef CRC was organised into the seven program areas listed in Table 1, together with the designated sets of activities undertaken to achieve the CRC’s overall objectives. The research program areas were used as the basis for budget allocations, project reporting and operational planning. The cash inputs to each of these seven programs were used as inputs to the Impact Tool.

Table 1.  Beef CRC programs and designated activities

To meet the additional input needs of the Impact Tool, which has a focus on the use of specific outputs to create outcomes and hence impacts, another way of reporting on CRC achievements was needed. Thus all of the individual planned outputs from the RD&E were grouped into 17 ‘products’ summarised in Table 2. Some of these products were closely related to a very specific set of activities in a particular research program. For example, one of the products is a DNA test for the polled gene in Australian beef cattle breeds, while another is a set of candidate antigens for a vaccine to control cattle ticks. Both of these products were derived from specific project areas. Other products however are an aggregation of outputs from a range of activities across several research programs. For example, one of the products is a set of genomic prediction equations for growth, feed efficiency, carcass and beef quality and reproductive performance. This product encompasses 12 separate outputs across four different research programs.

Table 2.  Product use and key end-users of Beef CRC products
P1, probability that all required outputs are produced to enable this usage. P2, probability of usage given required outputs have been generated

The Impact Tool aims to identify the potential impact of a funding proposal, or the realised and anticipated impacts from existing funding, by articulating the process by which research leads to impacts on the end-user and/or the broader community. The Cooperative Research Centres Association developed the initial version of the tool in 2007 and published a comprehensive discussion paper on the rationale behind impact evaluation modelling, together with some worked examples (CRC Association 2007). Recently, the CRC Program assumed responsibility and upgraded the tool (CRC 2013). One of the objectives of mandating the use of the Impact Tool is that a common framework is available to enable direct comparisons of multiple CRCs across diverse industries and discipline areas.

The Impact Tool uses an input-to-impact chain model involving a systematic grouping of different types of information. Fig. 1 is derived from the Impact Tool User Guide of the CRC Program Guidelines (CRC 2013) and sets out a schematic of the input to impact chain model. Several alternative documents have been made available, by a range of state government agencies, to provide guidance on how the Impact Tool should be used (e.g. Government of Victoria 2013).

Fig. 1.  The inputs to impact chain (Source: CRC Program 2013).

The Impact Tool software is written in Excel format and when first downloaded comes as a generic template. The user is required to specify the form of the template that suits their particular CRC. Embedded macros then format the Impact Tool to that specification. Thus, specifying seven research programs as in the Beef CRC results in seven sets of input, activity, output, usage and impact tables. For each of the seven programs the user is required to populate the relevant tables with the financial value of the resources used, the activities undertaken, the various outputs produced and so on. There is also an overview tab generated at the front that has seven tables like Fig. 1 which summarises the information for each program. The model requires costs and outputs to be specified annually over a 10-year period, usage to be specified over a 12-year period and impacts to be specified over a 15-year period. So starting at 2005/06, costs and outputs can be measured up to 2015/16, usage to 2017/18 and impacts to 2020/21. Wherever monetary values are inserted, the program automatically adds them over the relevant time frame and then discounts them at a standard rate (5%).

From the output and subsequent tabs, more specific detail is required on what is produced, how it is used and how it is valued. So the 17 separate CRC products described in Table 2 are allocated across the seven research programs. However as some products are combinations of outputs from several programs, and some products have different versions for different purposes, 28 separate outputs were specified (5, 3, 3, 2, 9, 1, 5 across programs 1–7 respectively). In this analysis each of these products had a separate usage and a separate impact to ensure nothing was missed. However, when specifying the Impact Tool it is possible for particular products to have multiple uses and multiple impacts, and vice versa.

For each product, the usage tab requires a description and justification of the expected annual adoption over a 12-year period, an estimate of any additional annual costs that users would have to incur for further refinement or application of the output, and estimates of the probability that the output would in fact be produced as planned and of the probability that usage will occur as anticipated. This is where knowledge of industry systems and institutions is important and where detailed studies of the new tools and technologies in the farm or value chain business are crucial.

Next, and again for each product, the impact tab requires a description and justification of the expected annual monetary benefit over a 15-year period, usually a per unit benefit multiplied by the usage numbers estimated earlier, and estimates of the probability that the usage and monetary impacts would occur as planned. Again, access to detailed studies of the implementation of new tools and technologies in the farm or value chain business are crucial. At the end of each program section in the impact tab, non-monetary impacts also can be specified.

A further section requires that the key risks which might prevent the outputs, usage and impacts occurring as anticipated are described. This is followed by the benefit: cost analysis tab where for each program the sum of the discounted benefit stream for all of the products specified in that program are set against the sum of the discounted R&D and usage costs for that program to generate a program specific benefit : cost ratio (BCR). Benefits and costs across all programs are then added to generate a whole of CRC BCR.

There is one final section called ‘additional information’ where detailed calculations for a specific product or products can be shown, and where relevant references can be cited.

Therefore what is being measured (and the type of data required) will depend on the life-cycle stage at which it is used. When it is used as part of an exit report at the end of the funding period, real data that have accrued over the life of the CRC will be used, together with projected data for the remainder of the 15-year period.

Many of the expected economic benefits from the Beef CRC’s third term were generally still to be realised at the end of the funding period (2011/12) as many of the products were still being delivered. These anticipated impacts were therefore assessed using estimates of production levels and costs and prices (at regional level where appropriate) derived from published analyses relating to each CRC product, with projections of future usage levels. The User Guide stresses the notion that the process is more important than the results: ‘The intent of the Impact Tool is not to generate precise estimates of all CRC costs and benefits, but rather to assist the consideration of the potential costs and benefits in a systematic way and to enable key issues that impact upon the potential projected value to be consistently addressed.’ (CRC 2013, p. 4).

That said, the best estimates of the usage and key endusers of each of the CRC’s products are reported in Table 2, and the best estimates of the per unit and aggregate monetary impacts of each of the CRC’s products are reported in Table 3. A listing of potential non-monetary impacts is given later.

Table 3.  Economic impacts of Beef CRC products

A summary of the economic benefits of the Beef CRC, as measured by application of the Impact Tool, is provided in Table 4. The estimates come from the benefit: cost analysis tab where for each program, the sum of the discounted benefit stream for all of the products specified in that program is set against the sum of the discounted R&D and usage costs for that program to generate a program-specific BCR. Benefits and costs across all programs are then added to generate a whole-of-CRC BCR over 15 years, starting at the first year of RD&E investment.

Table 4.  Return on investment metrics by program and whole-of-CRC as determined by the Impact Tool, 2005/06–2020/21

Behind these aggregate numbers lie detailed calculations that rely on the information provided in Tables 2 and 3. Let us take as an example the first product listed in Tables 2 and 3, the development of genomic prediction equations.

What we have to measure is the value of increased rates of genetic progress in the commercial beef herd due to incorporation of the new genomics information into EBVs. The base model comes from Banks (2012). This is a spreadsheet that maps historical and expected future rates of genetic progress in the Australian beef herd, weighted by breed types and markets, as valued by BreedObject (Barwick and Henzell 1998). Account is taken of the proportion of cows mated to BREEDPLAN registered bulls and the generation interval for cattle. Specific assumptions are then made about the influence of the genomic prediction equations on EBV accuracy and therefore on the nature and timing of changes in the rate of genetic progress. Given the information available in June 2012, we estimated that the rate of genetic progress would increase by $0.39/cow/year from 2016/17 to 2030/31 (Griffith 2012b). The relevant annual benefits were then transferred to the Impact Tool, where they are discounted, summed up to 2020/21 to give present values, and finally multiplied by the relevant probabilities (P3 = 87.5; P4 = 90). The net result is a net present value (NPV) of $12.4 million for the 4 years 2016/17–2020/21. This is the figure which appears in the right hand column of the first row of Table 3 and in the second last row of Table 4 against Program 7 benefits.

Thus, across all of the program areas, some $120 m (in 2005/06 dollars) was expected to be spent, either on the RD&E over the 7 years to 2011/12, or on extra costs expected to be incurred by users of the CRC products up until 2020/21. User costs might include the costs of DNA tests by individual producers or the costs of software packages such as the expanded versions of BeefSpecs (McKiernan 2011). Some $353 m of economic benefits (in 2005/06 dollars) were expected to be generated by using the CRC products up until 2020/21. These benefits included better compliance to market specifications, better rates of genetic gain, better nutrition, better management, etc, as detailed for each product in Tables 2 and 3.

Two commonly used measures of return on investment are the NPV and the BCR. The NPV is defined as the sum of discounted benefits minus the sum of discounted costs, or $353.1 m–$119.9 m = $233.2 m, evaluated over the period 2005/06–2020/21. Thus this investment, as measured by the Impact Tool, is expected to return a surplus of $233 m to the Australian beef industry over this period. The BCR is the sum of discounted benefits divided by the sum of discounted costs, or $353.1 m/$119.9 m = 2.94. Thus this investment is expected to return almost $3 for every $1 invested over the period 2005/06–2020/21. On both measures, investing in the Beef CRC is expected to have been profitable, and according to the User Guide, the calculated BCR fits very comfortably with the expectations of the Commonwealth Government for the net returns from CRCs.

The pattern of NPV and BCR across the seven programs is quite varied. This partly reflects the way the programs were initially set up and funded compared with the way the individual products were subsequently allocated to program areas. For example, program 6 (administration) incurred $14.1 m in discounted costs over the period 2005/06–2011/12 but had no outputs set against it. Conversely, program 7 (underpinning science) was set up mid-way through the CRC when the genomics technology that was the cornerstone of our RD&E strategy changed radically following the availability of the bovine genome sequence in 2005/06. It had no costs explicitly set against it but it was the home for most of the genomic products, valued at $12.4 m over the period 2005/06–2020/21. Further, programs 1 and 5 captured the benefits of some of the expenditure undertaken in programs 2 and 4 in relation to new and improved genetic parameters (program 1) and improved management tools and practices (program 5). Finally, the fact that program 5 generated a relatively large proportion of the benefits reflects the types of tools and technologies developed there compared with other programs. Most of the management tools and practices sitting in program 5 were already adopted by the cessation of funding, or were expected to be adopted in the near future, so there were quite a few years of realised or anticipated benefits able to be counted in the Impact Tool. However genetic technologies residing in programs 1 and 7 are slower to become available and accumulate gradually over time, so there are only a few years of realised or anticipated benefits able to be counted and the change in genetic gain is very small in those early years.

It was not possible to calculate a financial value of many of the Beef CRC’s outputs. For example, the value of an outcome such as ‘improved animal welfare’, depends on the preferences of both Australian and foreign consumers in a wide range of export markets who demand that animals be treated with utmost care, that their production environments remain ‘natural’, and that livestock remain ‘clean and green’. Little research in these areas has been done in Australia, so these types of benefits were classified as non-economic when completing the Impact Tool.

Similarly, the value of animal science education to the Australian economy is not readily quantifiable, although clear evidence exists that ongoing improvements in productivity are affected by levels of basic education and levels of continuous employee development. Over 40 post-graduate students were trained over the time period of the Beef CRC, with the majority being retained in Australia and in the livestock sector. As well, the CRC’s undergraduate and vocational training programs have provided enhanced technical and human capacity in industry and the wider economy.

Hence, in addition to the direct economic benefits accruing from the use of Beef CRC products, significant additional social, environmental and animal welfare benefits have accrued as a result of research outputs from the Beef CRC, including:

• Unique ability to guarantee the eating quality and food safety of beef, thereby increasing consumer confidence in Australian beef products;

• Improved on-farm and abattoir best practice to reduce pathogen loads on beef carcasses, increasing consumer confidence in beef safety;

• More skilled beef industry workforce across rural and regional Australia; and

• Increased commitment, loyalty and trust across beef value chain sectors arising from the collaborations forged through all three Beef CRC terms.

• Reduced methane emissions per kg of beef product arising from improved productivity of beef businesses and reduced feed requirements through improved feed efficiency;

• Confidence by consumers that Beef CRC products have been extensively developed and validated under Australian beef production systems and are therefore sustainable for Australian environments;

• Improved use of the pasture resource base through improved feed efficiency and reproduction; and

• Reduced chemical use in cattle production systems, reducing the risk of environmental contamination.

• New cattle welfare standards and guidelines underpinned by cattle welfare studies;

• Improved production, transport and pre-slaughter lairage systems designed to improve cattle welfare; and

• More resilient cattle through reduced stress and improved behaviour.

The outcomes and impacts achieved by the Beef CRC also resulted in several prestigious national and international awards, including (i) three separate Eureka Awards for Bioinformatics, Animal Welfare and Interdisciplinary Research; (ii) two CRC Association national Awards for Excellence in Innovation; and (iii) the International Meat Secretariat Millenium Prize for Meat Science and Technology.

The Beef CRC operated for its third successive 7-year term from July 2005 to June 2012. Some $120 m (in 2005/06 dollars) was spent in trying to achieve its objectives. Now that the funding period has ceased this analysis aims to determine whether this money was well spent.

It does this using the Impact Tool, which evaluates the impact of all the outputs relative to the input costs (cash and staff and non-staff in-kind contributions over the 7-year term of the CRC plus industry implementation costs), to estimate an overall BCR of the CRC over a 15-year time horizon. Based on several detailed economic analyses, estimates were made of economic benefits for most of the CRC’s 17 separate ‘products’, as summarised in Table 2 above. Some of these benefits have already been realised, however most of the benefits are those anticipated to accumulate over the next 7–8 years. The NPV, the sum of discounted benefits minus the sum of discounted costs, was estimated to be $233.2 m, while the BCR, the sum of discounted benefits divided by the sum of discounted costs, was estimated to be 2.94, both measures evaluated over the period 2005/06–2020/21. Thus on both measures, investing in the Beef CRC has been profitable.

In using the Impact Tool, several lessons have been learnt. First, the Impact Tool is an excellent vehicle for forcing the applicants for CRC funding, and then the managers of the expenditure if funding is secured, to have an explicit focus on achieving outcomes and impacts. While such a focus is mandatory within CRCs, and was always a priority within the Beef CRC, it is not a widely held view externally (Griffith 2008). We argue that the Impact Tool could have a wider role in RD&E funding and management more generally, such as in state and other federal government agencies, universities and other research organisations and in the rural industries’ RD&E corporations.

Second, using the Impact Tool for annual monitoring of progress against planned milestones provides an objective and effective mechanism for realigning the RD&E portfolio if that is required due to changing circumstances. The Beef CRC set very ambitious targets from the outset and we were very serious about managing the RD&E portfolio to try to achieve those targets. When the genomics technology that was the cornerstone of our RD&E strategy changed radically with the availability of the bovine genome sequence and associated new technologies in 2005/06 (see Goddard and Hayes 2009; Goddard 2010), the focus on achieving outcomes and impacts that is embedded in the Impact Tool allowed a rapid and effective realignment of the portfolio.

Third, the detail on per unit net benefits and on user numbers over time required to populate the Impact Tool provides explicit guidance on the variables that have to be measured and monitored over the life of the CRC and beyond, on the type of detailed economic evaluations that have to be done at the product level, and on the modelling frameworks that have to be used to support the targets and the measures of success.

However, using the Impact Tool is not always straightforward. First, one of the objectives of the CRC program in mandating the use of the Impact Tool is that a common framework was available to enable direct comparisons of multiple CRCs across diverse industries and discipline areas. Thus, ‘one size fits all’. However many of the agricultural CRCs are disadvantaged by this, especially those where investment in genetic improvement is a key aspect of the RD&E portfolio. The primary focus of the Beef CRC was new genetics and genomics technologies in beef cattle. Genetic improvement is a long and slow process, gradually accumulating over time. Due to the long generation intervals in cattle (~5 years), most genetic benefits accrue beyond a 15-year timeframe.

Second, the Impact Tool is essentially an accounting framework rather than an economic framework. Costs and benefits are described and quantified where possible, and discounting is used to bring all monetary values to a common point in time, but there is no mechanism in the Impact Tool for economic incentives to trigger responses by producers or consumers. Depending on the industry and the time frame under consideration, this may mean a significant set of benefits are not counted in the Impact Tool. If the user wishes to properly account for these extra benefits, additional analyses have to be done either before data are entered, or preferably, afterwards. Additional analyses also have to be done if the analyst is interested in the distribution of benefits across segments of the value chain, including consumers domestically and in export markets, or across different production regions.

Finally, the Impact Tool makes no attempt to examine the ‘with-CRC’ scenario relative to a ‘without-CRC’ scenario: what would have the investment been and what would have the outcomes and impacts been if the CRC had not been funded? This is especially relevant to this CRC as much of the new and improved genetic information available from the CRC’s RD&E would be implemented via existing industry schemes such as BREEDPLAN to ensure easier use and greater understanding by industry about how best to apply the new technologies to achieve value. While this makes adoption easier, it creates potential difficulties in properly attributing changes in rates of genetic progress to CRC investment or to other past or new investments outside of the CRC.

However, if a multi-pronged approach to impact assessment is envisaged, these disadvantages can be overcome. In particular, the Impact Tool provides many of the inputs and assumptions on each of the CRC’s products necessary for a third, more aggregate but more economic, level of evaluation. That third level uses the DREAM benefit: cost analysis program (Wood et al. 2001) to provide a direct assessment of progress towards the CRC’s overall targeted outcome as recorded in the Commonwealth Agreement. It evaluates the CRC’s realised and potential impacts over a 25-year timeframe, which is more appropriate for beef genetic technologies. In addition to domestic industry benefits, it also includes wider benefits such as those achieved by export markets and consumers and enables a ‘with-CRC’ and ‘without-CRC’ comparison to be made. Application of this framework is reported in Griffith et al. (2013).

In summary, in our experience, the value of the Impact Tool is not only for ex-ante and ex-post-evaluation of the impacts of particular technologies, but it also provides a very effective tool for RD&E project planning. By way of example, we have found that investing in learning and using the Impact Tool:

• Ensures the products or outputs of the proposed research are clearly identified and targeted in all planning from the outset;

• Enables development of a clear ‘path to adoption’ for every product on a customised product-by-product basis, including clearly defined end-users and an understanding of how those users will use or adapt the product for use in their own businesses;

• Provides guidance and a ‘reality check’ on the effectiveness of the proposed and alternative commercialisation models for each product;

• Ensures the costs of implementation of research results in commercial businesses are not overlooked or inadvertently ignored;

• Provides a mechanism so that the probabilities of success and adoption are challenged and cross-checked for accuracy, providing an additional ‘reality check’ on individual components of the entire commercialisation and adoption processes for each product; and

• Allows effective evaluation of alternative research strategies when developing new RD&E project proposals, in both developed and developing countries.