Evaluation of ACCESS-S1 seasonal forecasts of growing season precipitation for Western Australia’s wheatbelt region

2.1. Data

2.2. Analysis

Two forecast periods during the growing season were selected for analysis based on the timing of key rainfall-related management decisions. One of the most important management decisions involves choosing how much fertiliser to apply so that favourable conditions are either exploited to maximise profits, or resources are conserved during less favourable conditions to minimise losses. Fertiliser is initially applied at the beginning of the growing season based on the rainfall outlook for the season ahead. Decisions are then made as to whether to apply a ‘top up’ of fertiliser approximately mid-way through the season based on rainfall that has already fallen and the outlook for the remainder of the season. To tie into these two critical decision-making time periods, this study evaluated May–October (MJJASO) and July–September (JAS) precipitation forecasts with both 0- and 1-month lead times.

The forecasts were analysed based on wet, dry and average rainfall years. To classify years as wet, dry and average, the SWWA domain (36–28°S; 114–122°E; Fig. 2) averaged anomaly for the MJJASO growing season (GS) was calculated from the AWAP observational data for each year. Based on an evaluation of all of these values together, a subjective decision was made to use the anomaly thresholds as follows: <−40 mm per GS = dry year, >40 mm per GS = wet year and >−40 mm per GS and <40 mm per GS = average year. This resulted in 6 dry years (2000, 2002, 2006, 2007, 2010 and 2012), 6 wet years (1992, 1996, 1998, 1999, 2005 and 2011) and 11 average years (1990, 1991, 1993, 1994, 1995, 1997, 2001, 2003, 2004, 2008 and 2009) for analysis. We note that because years are classified as either wet, dry or average based on the entire growing season, for some years, the JAS anomaly may not necessarily show a positive (negative) anomaly for a wet (dry) year. Although it could have been possible to also use the JAS anomaly to classify years, this would have resulted in a different set of years being classified as wet, dry and average for JAS compared with MJJASO. Because the interest is largely in whether a particular growing season will be either very wet or very dry overall, for simplicity we just used the anomaly for the MJJASO period to classify years.

Fig. 2. 

The SWWA domain used for analysis showing ACCESS-S1 topography.

Anomalies for the ACCESS-S1 hindcasts were calculated by computing the climatology (1990–2012) from the ensemble mean hindcasts for each start date. The climatology was then subtracted from the ensemble mean (or individual ensemble members) for each start date to produce the forecast anomalies, and in so doing a first-order linear correction for model bias or drift was made (Hudson et al. 2017b).

The AWAP dataset was used as a reference to estimate the errors in ACCESS-S1 GSP, and the ERA5 dataset allowed an exploration of the dynamical processes driving the errors. We examined the ACCESS-S1 ensemble mean precipitation during the growing season spatially over the SWWA domain and compared it with the AWAP observations, for all years, as well as wet, dry and average years. We also assessed the ensemble spread, displayed as box and whisker plots for the SWWA region.

To contribute to an understanding of the model errors in rainfall over SWWA, the ability of ACCESS-S1 to capture the frequency and timing of rain-days of different intensities, i.e. light to heavy rainfall events, was evaluated. Five rain-day categories were defined based on the 25th, 50th, 75th and 95th percentiles for the daily domain sum of rainfall from ACCESS-S1 and AWAP (Table 1). ACCESS-S1 had lower rainfall values than AWAP across all percentiles (Table 1). The percentiles were used to generate five rain-day categories (Table 2). We examined the frequency of different rain-day categories in ACCESS-S1 and AWAP for wet, dry and average years. Additionally, we examined the timing of the most extreme rainfall (i.e. categories 4 and 5; Table 2), as these heavy rainfall events can have significant impacts. The rain-day category analysis was based on the entire growing season and 0-month lead time.

The final part of the analysis focused on the large-scale circulation, including SST and wind anomalies, and how well these were captured by ACCESS-S1 during wet and dry years.

3.1. Spatial analyses of mean bias

The first stage of our analysis involved evaluating the ensemble mean bias in ACCESS-S1 spatially across SWWA for the full growing season of MJJASO as well as for the JAS forecast period for lead times of 0- and 1-month. This allowed us to initially assess how the model errors varied regionally. Fig. 3 shows the total precipitation for JAS and MJJASO averaged over 1990–2012 for the AWAP observations (re-gridded to ACCESS-S1 resolution) and for the ACCESS-S1 ensemble mean at 0- and 1-month lead times. The observations showed that most rainfall occurred in the south-west corner and there was a clear south-west to north-east gradient with rainfall decreasing moving further inland. Although ACCESS-S1 captured this rainfall pattern well, rainfall in the south-west corner was markedly underestimated, with a notable dry bias evident. This was apparent in both forecast periods and there was no obvious drift in the bias with lead time. These findings are consistent with those of Hudson et al. (2017a), who showed that although the rainfall climatology in ACCESS-S1 is much improved compared to POAMA, its predecessor, there is still a dry bias over SWWA.

Fig. 3. 

Total precipitation averaged over 1990–2012 from observations (left panels) and ACCESS-S1 with 0-month (middle panels) and 1-month lead times (right panels) for (a) July–September and (b) May–October.

Similarly, Fig. 4–6 show the precipitation anomalies for dry, wet and average years respectively (refer to Methods for definitions of wet, dry and average). For dry years (Fig. 4), although the ACCESS-S1 anomalies were negative in some areas of SWWA, the forecasts did not capture the significant deficiency of rainfall for these years. The ACCESS-S1 forecasts suggested closer-to-average conditions rather than the dry conditions that were observed. This bias appeared more pronounced for the 1- compared with the 0-month lead time particularly for the MJJASO forecast (Fig. 4b). ACCESS-S1 better captured the wet years compared to dry years in that the precipitation anomalies were largely positive across most of SWWA (Fig. 5). However, the forecasts still showed a dry bias, slightly more evident at 1- compared to 0-month lead time. During average years, forecasts at 0-month lead time showed slightly drier conditions whereas the forecast at 1-month lead time showed average to neutral conditions (Fig. 6).

Fig. 4. 

Precipitation anomalies using a 1990–2012 climatology averaged across dry years for observations (OBS) and ACCESS-S1 with 0- and 1-month lead time for (a) July–September and (b) May–October.

Fig. 5. 

Precipitation anomalies using a 1990–2012 climatology averaged across wet years for observations (OBS) and ACCESS-S1 with 0- and 1-month lead time for (a) July–September and (b) May–October.

Fig. 6. 

Precipitation anomalies using a 1990–2012 climatology averaged across average years for observations (OBS) and ACCESS-S1 with 0- and 1-month lead time for (a) July–September and (b) May–October.

The results in Fig. 4–6 demonstrated an overall poor performance of ACCESS-S1 in simulating GSP for SWWA. Having identified the bias in the ensemble mean forecasts, we took a more probabilistic approach, and examined the ensemble spread for all years combined as well as each individual year, to provide an assessment of forecast skill.

3.2. Ensemble spread and forecast skill

The ensemble spread for the SWWA region is illustrated in Fig. 7–9. These show the median and spread of the ACCESS-S1 ensemble precipitation anomalies averaged across SWWA compared with observed for dry, wet and average years respectively. The JAS and MJJASO forecasts are shown for both 0- and 1-month lead times. All years are plotted together in panel (a) and each year is plotted separately in panel (b) to provide an indication of forecast skill.

Fig. 7. 

Median and spread of ACCESS-S1 ensemble anomalies averaged over SWWA compared with observed anomaly (red dots) for 0- and 1-month lead times and for July–September (JAS) and May–October (MJJASO) forecasts for (a) all dry years combined and (b) each dry year (boxes show 25th to 75th percentiles, whiskers show Q1 − 1.5 × IQR and Q3 + 1.5 × IQR, outliers (open circles) show >1.5 × IQR).

Fig. 8. 

Median and spread of ACCESS-S1 ensemble anomalies averaged over SWWA compared with observed anomaly (red dots) for 0- and 1-month lead times and for July–September (JAS) and May–October (MJJASO) forecasts for (a) all wet years combined and (b) each wet year (boxes show 25th to 75th percentiles, whiskers show Q1 − 1.5 × IQR and Q3 + 1.5 × IQR, outliers (open circles) show >1.5 × IQR).

Fig. 9. 

Median and spread of ACCESS-S1 ensemble anomalies averaged over SWWA compared with observed anomaly (red dots) for 0- and 1-month lead times and for July–September (JAS) and May–October (MJJASO) forecasts for (a) all average years combined and (b) each average year (boxes show 25th to 75th percentiles, whiskers show Q1 − 1.5 × IQR and Q3 + 1.5 × IQR, outliers (open circles) show >1.5 × IQR).

Overall, Fig. 7–9 show that regardless of whether the season was very wet or very dry, the ACCESS-S1 ensemble median forecast anomaly was generally weak. This is consistent with our spatial analysis of the ensemble mean (Fig. 4–6) that showed close to average mean forecasts during both dry and wet years. For dry years, the ACCESS-S1 ensemble spread generally captured the observed precipitation anomalies, but mostly within the lower tail of the distribution, explaining why the ensemble mean precipitation for dry years was not dry enough (Fig. 4). For the JAS forecasts, the ensemble median anomaly was closer to the observed anomaly compared with the MJJASO forecasts for most years, with the exception of 2010 and 2012. Notably, these were both particularly dry years. We note that all years were classified as wet, dry or average based on the precipitation anomaly of the MJJASO period. Hence, for some years, the observed JAS rainfall anomaly for the dry or wet years may be closer to zero if these anomalies happened predominantly during May–June or in October.

The years when the observed anomalies were most negative, i.e. the driest years (2006, 2010 and 2012) were the years when the ACCESS-S1 ensemble spread least consistently captured the observed anomaly. For example, in 2010, one of the driest years on record, none of the forecasts for either time frame or lead time captured the observed anomalies in its ensemble spread. Furthermore, for the MJJASO forecast at 1-month lead time, the ensemble median actually showed a forecast with a positive anomaly. This implies that ACCESS-S1 had limited skill in capturing more extreme conditions. When examining all years together, the ensemble spread increased with lead time for dry years. However, this pattern was not consistent for each individual year. For example, for the year 2000 the ensemble spread decreased with increasing lead time for both the JAS and the MJJASO forecast periods.

For wet years (Fig. 8), the opposite was shown, with the observations mostly in the upper tail of the ensemble spread. We note that the observed anomalies for some years for the JAS period were close to zero or negative, suggestive of average or dry conditions. The classification of those years being wet was due to the wetter-than-average conditions experienced in May–June or October rather than the JAS period. ACCESS-S1 performed marginally better for wet years compared to dry years in that the average anomaly was generally positive across the region. When examining the forecasts for all wet years, for the MJJASO forecast period, the observations were mostly within the upper tail of the ensemble spread. This can explain why ACCESS-S1 overall did not simulate large enough positive precipitation anomalies during wet years, as illustrated in Fig. 5. For the 1-month lead time for MJJASO, it is worth noting that the spread in ACCESS-S1 was much larger above the median and the observations generally lay in the lower part of the upper tail. The larger upper tail in the ensemble spread for the 1-month lead time forecast shows that there was a large range of values among the higher rainfall forecasts and hence less agreement among ensemble members. When examining all wet years, the ensemble spread decreased for increasing lead time for the JAS forecast but increased for the MJJASO forecast. It is expected that with increasing lead time there would be more uncertainty with a forecast and increased spread, as demonstrated for MJJASO, and it may be that the spread difference between the two lead times for JAS was not significantly different. For wet years (Fig. 8b), forecast skill appeared poorest during the wettest years (1992, 1996 and 1999), supporting the premise that ACCESS-S1 is limited in its forecasts of extreme rainfall in this region and time of year. Notably, for 1992 and 1996, when a significant proportion of the rainfall occurred during the JAS period, the observed anomaly was still not captured within the ensemble spread even for the shorter forecast period of 3 months and 0-month lead time.

For average years (Fig. 9), the observed anomalies fell within the middle range of the ensemble spread, which is to be expected since ACCESS-S1 tends to forecast average conditions for these forecast periods and lead times as shown earlier. Across all years, the ensemble spread increased slightly with increasing lead time for the JAS forecast but decreased for the MJJASO forecast.

The results above (Fig. 7–9) showed that model skill was limited, especially for very dry and very wet years. Previous studies that investigated seasonal precipitation forecasts in ACCESS-S1 have focused on periods of 1 month (e.g. King et al. 2020) and up to a 3-month season (e.g. Hudson et al. 2017b; Marshall et al. 2021a). King et al. (2020) examined the area-averaged correlation coefficient matrices for each state of Australia for each calendar month and for both 0- and 1-month lead times. Relevant to the present study, the correlation for total precipitation averaged across WA at 0-month lead time with observed, was statistically significant for forecasts for May, July, August and September. However, with 1-month lead time, none of the monthly forecasts had significant correlation coefficients for WA, demonstrating that forecasts beyond the first month performed poorly for the broader region. Similarly, Hudson et al. (2017b) evaluated ACCESS-S1 skill for forecasts of four separate seasons of 3 months duration for the key horticultural regions across Australia at both 0- and 1-month lead times. For the SWWA region, there were no significant correlations with observations for any of the forecasts (refer to fig. 2, A1 and A2 of Hudson et al. 2017b). Marshall et al. (2021a) examined correlations between the MJO and extreme precipitation across Australia. For SWWA during strong phases of the MJO, they showed some skill during SON and DJF for small parts of SWWA (fig. 11 of Marshall et al. (2021a)), but not during MAM and JJA, which are the key seasons during the growing season. Our results demonstrated that forecasts for a longer period, i.e. a 6-month growing season, were similarly limited in terms of skill.

3.3. Rain-day category frequency and timing based on percentiles

We next investigated how accurately ACCESS-S1 captured the frequency and timing of light to heavy rainfall events (rainfall was aggregated over the SWWA domain, i.e. gridboxes receiving rainfall were summed to create the domain total) to understand how these variables contributed to the inaccuracies in the ACCESS-S1 GSP forecasts. Fig. 10 shows the relative frequency of the different rain-day categories for ACCESS-S1 and AWAP for dry and wet years. For dry years, ACCESS-S1 overestimated the frequency of categories 4 and 5 but underestimated the driest category (rainfall less than the 25th percentile). The overestimation of the heavier rainfall categories contributed to ACCESS-S1 not sufficiently capturing the dry conditions during dry years (Fig. 4 and 7). For wet years, ACCESS-S1 had a very similar rainfall distribution to AWAP, perhaps slightly underestimating the heaviest rainfall categories.

Fig. 10. 

Relative frequency of rain-day categories (Table 2) for (a) dry years and (b) wet years for ACCESS-S1 at 1-month lead time and AWAP (error bars for ACCESS-S1 denote the standard deviation of the ensemble mean for each category).

We next examined the timing of the heavy rainfall events showing the relative frequency of rain-day categories 4 and 5 for ACCESS-S1 compared with AWAP for dry and wet years for each month of the growing season (Fig. 11). For dry years, it is most notable that ACCESS-S1 forecast too many extreme (category 5) rainfall days in May and June but too few for July–October. The differences between ACCESS-S1 and AWAP were less marked for the timing of category 4 rain-days. Category 5 rain-days represent extreme rainfall events which, in this region, are most commonly driven by strong cold fronts traversing the region during winter (Hope et al. 2014). These results therefore suggest that during dry years, ACCESS-S1 likely simulated the intense cold fronts that bring the heavy rainfall days in SWWA too early in the season. Since these mid-latitude fronts are not strongly directly linked to any one mode of natural climate variability, this inevitably makes their predictability on seasonal timescales more challenging. King et al. (2020) showed in their evaluation of ACCESS-S1 that model performance was superior in the northern areas and the interior where climate modes of variability such as ENSO have a greater influence in comparison to the south and coastal regions. Nonetheless, as far as we are aware, this inconsistency in the timing of extreme rainfall events in SWWA has not been identified in previous evaluations of ACCESS-S1 and should be explored further when developing and improving the model.

Fig. 11. 

Relative frequency for each month of the growing season (May–October) for dry years (a) category 4 and (b) category 5 rain-days and wet years (c) category 4 and (d) category 5 rain-days for AWAP and ACCESS-S1. Error bars for ACCESS-S1 denote standard deviations for ensemble means.

For wet years, there was a similar pattern of differences between ACCESS-S1 and AWAP for category 5 rain-days (Fig. 11b). Early growing season heavy rainfall was overestimated (May–July) and later in the season was underestimated (August–October). It is likely that during wet years too many cold fronts were simulated early in the season but too few later in the season.

The analysis of the frequency and timing of low to heavy rainfall events was based on the domain sum of rainfall across the south-west region. It is acknowledged that the majority of this rainfall comes from the south-west coast (Fig. 3) and is therefore outside of the agricultural region which is the main focus of this study and by using the domain sum the spatial differences were lost. We re-performed the analysis but focused on a smaller region of the south-west (36–28°S; 114–117.5°E), where the majority of rainfall occurs. The results obtained (not shown) were very similar to Fig. 10 and 11, demonstrating that these results were largely driven by the rainfall from this smaller region in the south-west corner. This highlights a limitation due to the relatively coarse 60-km resolution of ACCESS-S1 when examining a small region like SWWA. Higher spatial-resolution seasonal forecasts in the future would allow a better understanding of rainfall biases within the inland wheatbelt region. However, given that the main rain-bearing frontal systems would also bring rainfall inland, although with lower rainfall amounts, it is not unreasonable to suggest that a similar distribution frequency would be obtained if the percentile thresholds were relative to the inland regions.

3.4. Large-scale circulation

It was established in previous work that variations in Indian Ocean SST and surface winds can partly explain very dry–wet years in SWWA (e.g. England et al. 2006; Ummenhofer et al. 2008). Fig. 12 shows wind vector and wind speed anomalies over the growing season averaged across dry years and wet years for ERA5 compared with ACCESS-S1 at 0- and 1-month lead times. It is clear from ERA5 that during dry years, there is anomalous strong offshore (easterly–south-easterly) anticyclonic circulation across SWWA, suggestive of the drier conditions that were observed. The wind pattern identified here during dry years is similar to previous results (fig. 4 of England et al. 2006) despite the different reanalysis (NCEP v. ERA5 re-analysis) and timeframes (yearly v. growing season) used for the analysis. It is evident that ACCESS-S1 failed to capture this wind pattern driving dry conditions in the region. Instead, ACCESS-S1 simulated anomalous southerly–south-westerly winds moving across the region at 0-month lead time and weaker south-easterlies at 1-month lead time. During wet years, ERA5 showed an anomalous strong onshore (north-westerly) cyclonic circulation across SWWA, which would advect more moist air into the region, again very similar to results in England et al. (2006). There was some cyclonic motion evident in ACCESS-S1 at both lead times over SWWA; however, this was weaker than in ERA5. In ERA5 there were also strong northerly anomalies extending from the Maritime continent region and eastern tropical Indian Ocean down over WA, which were not captured in the model. Additionally, the anomalously strong wind speeds present north of Papua New Guinea and south-west of Sumatra during wet years were absent in the ACCESS-S1 simulations.

Fig. 12. 

Ten-metre wind speed and direction anomalies (reference period 1990–2012) averaged for May–October for (a) dry years and (b) wet years, for ERA5 and ACCESS-S1 at 0- and 1-month lead times. Vectors indicate direction only, with the wind speed anomalies shown as colour shading.

Fig. 13 shows SST anomalies averaged for May–October for ERA5 compared with ACCESS-S1 at 0- and 1-month lead times averaged for dry and wet years. England et al. (2006) identified a dipole of SST anomalies associated with wet and dry years for SWWA. This dipole has peak amplitudes in the eastern Indian Ocean adjacent to the west coast of Australia – off the north-west and south-east coast (fig. 5 in England et al. 2006). The dipole was also evident here in the ERA5 SST (Fig. 13), but with differences in the magnitude and spatial distribution due to different models, SST datasets and timeframes. During dry years there were anomalously cooler temperatures to the north of Australia and extending to Sumatra similar to the SST pole (P1) shown in England et al. (2006) (their fig. 5a) but evidently much weaker here compared to their study. Another pole of warmer temperatures was shown in the Indian Ocean to the west of WA, similar to the second pole (P2) identified by England et al. (2006). These poles switched sign during wet years; however, it is notable that, although there is symmetry between the wet and dry years in England et al. (2006), the dipoles identified in our study were asymmetrical. As described by England et al. (2006), this dipole in SST develops as a result of the anomalous wind fields in the region during extreme rainfall years. In addition, in dry years the SST anomaly in the tropics shows another dipole − a weak positive IOD pattern. The IOD is the SST anomaly difference between the western (50–70°E; 10°S–10°N) and eastern (90–110°E; 10°S–0°) tropical Indian Ocean (Saji et al. 1999). In ACCESS-S1 dry years, the anomalously cooler waters to the north of Australia were absent for both lead times, with the model simulating warmer than average SST. In addition, the tropical west–east SST anomalies in the model are indicative of a negative (0-month lead) or neutral (1-month lead) IOD.

Fig. 13. 

SST anomalies (reference period 1990–2012) averaged for May–October for dry years (top row) and wet years (bottom row), for ERA5 (left) and ACCESS-S1 at 0- (middle) and 1-month (right) lead times.

During wet years there are warm SST anomalies surrounding much of Australia, particularly the north and north-west, but there are cold SST anomalies over the central Indian Ocean and the signature over the tropics is for a negative IOD. ACCESS-S1 captured the general large-scale pattern of SST anomalies, but the warm anomalies north of Australia and the central Indian Ocean cold anomalies were much weaker than observed. The limited skill of ACCESS-S1 to capture the large-scale circulation and SSTs associated with wet and dry years in SWWA likely contributed to the errors identified in our analysis of GSP forecasts during the wheat growing season in SWWA.

These results highlight how errors in ACCESS-S1’s simulation of the Indian Ocean SSTs and wind fields potentially drive the shortcomings in SWWA rainfall prediction. Previous work also highlighted that model errors in simulating the relationship between Indian Ocean SST and Australian weather are potentially causing issues in forecasting for Australia. For example, Lim et al. (2016) demonstrated how the inability of ACCESS-S1 to accurately simulate the teleconnection between the tropical western Indian Ocean SST and Victorian rainfall limited the skill for rainfall forecasts in the region. Similarly, Marshall et al. (2021b) showed that, although ACCESS-S1 was able to capture the relationship between the negative IOD phase and extreme fire weather over Australia quite well, there were clear inaccuracies in the depiction of this relationship during the positive phase.