Validation of BARRA2 and comparison with MERRA-2 and ERA5 using historical wind power generation

1.Introduction

Wind power is a critical renewable technology and will need to expand significantly to meet climate mitigation goals (International Energy Agency 2021). However, wind is highly variable, both across space and across time. At sub-daily intervals, this variability presents system-level challenges when renewable penetration is high (Pichault et al. 2021). Understanding the characteristics of the wind resource across different regions and time horizons – both short and long-term – is essential. Long-term simulation or optimisation of these systems is often conducted using energy system models. One approach, known as bottom–up or engineering modelling, uses time series power datasets (or ‘traces’) as model inputs (Wang et al. 2018), which are sometimes derived from atmospheric reanalyses such as ERA5 (ECMWF Re-Analysis, fifth generation; ECMWF, European Centre for Medium-Range Weather Forecasts) or MERRA-2 (Modern-Era Retrospective analysis for Research and Applications, V2).

Several methods are available to produce hourly wind power traces. The default method, when suitable data are available, is historical time series output from operating wind farms. The strength of these is that they integrate the various important factors, including the nature of the site-specific wind resource, the wind power technology used to convert the kinetic energy of wind into electrical energy, and the various localised complexities, such as turbulence and topography. A weakness is that these traces pertain only to those wind farms, or prospective locations within the same wind regime. Wind regimes refer to regions in which wind speed is highly correlated. In general, correlation diminishes with widening distance between locations (Huva 2014). A further weakness is that traces can be produced only for the period for which the wind farm has been operating. There may also be factors that influence reported wind power such as outages and curtailment.

Another method is synthetic traces. Their strength is that they can be produced with prescribed statistical properties for any area over any time horizon (Sturt and Strbac 2011). Once a synthetic model is built, the parameters can be readily calibrated to match known performance across regions, such as wind conditions, terrain type and turbine model. Nonetheless, synthetic wind power traces are still dependant on historical data for calibration.

A further method is applying a power curve to wind speed measured at meteorological stations (Cowin et al. 2023). Wind speed is reliably measured with high temporal resolution at many locations, potentially significantly expanding the geographic coverage of wind power traces. The main weakness is that the standard meteorological mast height of 10 m is well below that of the typical wind turbine hub height of 80 m or higher. (Australian weather stations use a standard anemometer height of 10 m. Methods of sampling wind are given in Australian Standard AS/NZS 3580.14.) Wind observations may be influenced by local topography, vegetation and buildings. Wind speed sampling at 10 m may capture broad wind patterns, but is likely to be unrepresentative of wind power at typical wind turbine hub heights, even if adjustments are made for height. Average wind speed in undisturbed flow increases with height in an approximately logarithmic manner. However, when the flow is disturbed owing to changes in the ground roughness, the wind speed changes in a more complex manner (Standards Australia 2014).

An increasingly popular method of producing wind power traces is atmospheric reanalyses. Atmospheric reanalyses combine a climate forecast model with observations that are distributed irregularly in space and time into a spatially complete gridded meteorological dataset (Rienecker et al. 2011). Reanalyses include many variables, such as temperature, atmospheric pressure and wind speed. Observations come from a wide variety of sources including land-based weather stations, buoys, radiosonde balloons and satellites. Reanalyses span the historic weather record and are widely used in climate and weather research. Reanalysis data are freely available as geographically gridded data with hourly time step.

Reanalyses report wind speed at various heights, from which wind power can be estimated. Given a wind power trace dataset spanning a full year, the annual capacity factor can be determined for that year. The popularity of reanalyses stems from their convenience, long time spans and wide coverage (Staffell and Pfenninger 2016). The most prominent global reanalyses are NASA’s MERRA-2 (Rienecker et al. 2011) and the ECMWF’s ERA5 (Hersbach et al. 2023). In addition to these global reanalyses, several regional reanalyses are produced, including the Australian Bureau of Meteorology’s BARRA (Bureau of Meteorology Atmospheric Regional Reanalysis for Australia) reanalysis (Su et al. 2019), and the updated BARRA2 reanalysis (Su et al. 2022), which is the focus of the present study.

Although widely used for wind power modelling, the most prominent global reanalyses, MERRA-2 and the newer ERA5, are known to exhibit significant bias. In a validation of MERRA-2 covering 23 countries in Europe, Staffell and Pfenninger (2016) found that a simulation without bias correction resulted in an annual capacity factor error range of −30% up to +60%. They proposed a bias correction method based on a national-level capacity factor, which improved the correlation of simulated with observed wind power to above 0.95 at a national level.

A few studies have compared the accuracy of MERRA-2 and ERA5 for simulating wind power production. Jourdier (2020) assessed both reanalyses using France as a case study. Jourdier’s study had two components – a comparison of simulated with measured wind speed at eight sites, at mast height between 55 and 100 m, and a comparison of simulated wind power with observed wind power at wind farms. They found that MERRA-2 exhibited significant biases in annual capacity factor, and that ERA5 performed better, although ERA5 underestimated wind speed and exhibited problems at certain times of the day. Olauson (2018) assessed MERRA-2 and ERA5, both for aggregated national generation for five countries (Germany, Denmark, France, Sweden and Bonneville Power Administration in north-west USA), and also for individual wind turbines in Sweden, and also concluded that ERA5 performed better. They noted that the effects of local topography on wind speed could not be discerned in either MERRA-2 or ERA5, and advised against their use for determining average wind speed at a particular site. Using national-level data, they applied a correction method that involved linearly scaling the mean wind speed from the model so that the capacity factor of the model matched the measured capacity factor for the region under investigation. Gruber et al. (2022) assessed MERRA-2 and ERA5 in four countries (US, Brazil, South Africa and New Zealand), and assessed bias correction using the Global Wind Atlas, version 2 (GWA2). They found that ERA5 performed better than MERRA-2 and that error was not improved with bias correction. The overall median correlation was 0.82 for ERA5 and 0.77 for MERRA-2, and with a larger spread of correlations for MERRA-2. However, Murcia et al. (2022) found that bias correction of ERA5 using GWA2 showed good agreement with observed country-level generation data for Europe.

In Australia, only a few studies have used the BARRA reanalysis, and only one study could be identified that has assessed its accuracy in the context of wind power.¹ Cowin et al. (2023) compared the performance of BARRA-R (from the first version of BARRA) with MERRA-2 and ERA5 for offshore wind development and found a wide variance in simulated capacity factor between the reanalyses. In their study, reanalysis data were compared with weather station data at 10-m mast height.

ROAM Consulting used the forerunner of BARRA (ACCESS-A, Australia Community Climate and Earth System Simulator) to produce wind power traces for the Australian electricity system operator’s 100% renewables study in 2012 (Cutler et al. 2012). The study covered the eastern states, which form the National Electricity Market (NEM). ROAM Consulting used a generic power curve for all sites. Bias correction involved applying a scaling factor to wind speed so that the simulated capacity factor matched the known capacity factor at wind farms. The same data were recently used in an Australia renewables simulation undertaken by Rispler et al. (2022) to assess the potential role of offshore wind power in Australia’s future energy mix. A weakness of adjusting wind speed based on known wind power capacity factors after the fact is that it is unclear how accurate this adjustment will be for future wind farms. This is because the physical basis for such scaling is not well understood.

In another study covering the Australian NEM but using MERRA-2, Gilmore et al. (2022) assessed the risk of ‘energy droughts’, which are a concurrent lack of wind and solar over a given interval. Similarly to ROAM Consulting, bias correction involved applying a wind speed scaling factor to match the known capacity factor at wind farms. For new sites, a region-wide scaling factor was applied. The duration of energy droughts is a critical determinant of renewable firming and energy storage capacity. Laslett et al. (2016) also used MERRA-2 to generate synthetic time series wind power for the south west region of Western Australia.

The present study compares simulated wind power using the BARRA-R2 (Regional-scale reanalysis of BARRA, version 2) reanalysis (12- × 12-km grid) from the most recent BARRA2 suite, and compares the results with MERRA-2 (55- × 70-km grid) and ERA5 (30 × 30 km grid). These power estimates are compared with observed power at operating wind farms based on reported power from the electrical system operator for the two largest interconnected systems: the NEM and the South-West Interconnected System (SWIS). A total of 54 sites, corresponding to operational wind farms, are assessed. Data are included for 2014–2021, giving up to 8 years of data for some sites, based on data quality and availability.

Most studies assess wind power at a national or regional level, and many use generic turbine power functions. This study differs from previous studies in comparing simulated with observed power at the level of a wind farm, using site-specific turbine parameters.

Our objective is to assess the accuracy of the BARRA-R2 reanalysis for wind power modelling in Australia. It is essential to understand not only the bias in modelled wind power but also the confidence in this bias across regions where calibration data are not available. Observed wind farm power data provide the most reliable benchmark for energy system modelling and serve as a baseline reference. In addition to BARRA-R2, we evaluate the performance of MERRA-2 and ERA5, two of the most widely used reanalyses internationally. Although these reanalyses have been evaluated in other contexts, their accuracy in Australia remains less well understood. The metrics assessed include correlation, bias, power distribution and the hourly rate of change in simulated wind power. A further aim is to identify the factors influencing the accuracy of simulated wind power at the scale of a wind farm. These factors include technical aspects related to wind turbines and wind farms, as well as characteristics specific to the reanalyses.

The paper is structured as follows: Section 1 introduces the study and reviews the literature. Section 2 discusses the reanalyses, comparison methodology and analysis methodology. Section 3 presents the results and Section 4 discusses the results. Section 5 concludes the paper.

2.Data and methods

2.2. Electricity regions

Australia has two major grids, which cover most of the population, and many minor grids covering remote and regional communities. The largest grid by population and energy is the NEM, which includes the states of South Australia, Victoria, New South Wales, Queensland and Tasmania. Annual traded energy is currently 189 TWh (Australian Energy Regulator 2023). The second largest grid by energy is the SWIS, which covers the southwest of Western Australia. Annual traded energy is currently 20 TWh (Australian Energy Market Operator 2025). The present study includes sites located within both the NEM and the SWIS. All states were included except for the Northern Territory (see Fig. 1).

Fig. 1.

Geographic location of wind farms. For details of farms, see Tables A1 and A2 in the Appendix.

2.5. Wind power model

A wind power model was used to estimate wind turbine power, given wind speed, and modified by air density and a scaling factor that accounted for factors that reduced available power. For the purpose of this study, it is assumed that a wind farm behaves as a scaled-up, discrete wind turbine. Limitations of this assumption are discussed in Section 4.

The power available from a wind turbine is given by:

(1)P=Cp12ρAU3

where C_p is a variable turbine power coefficient, ρ is the density of air, A is the swept area and U is the magnitude of the wind speed. In typical wind turbines, C_p increases from ~0.3 at the cut-in speed, up to a peak of ~0.45 at 7.5–8.5 m s⁻¹ and decays by U⁻³ thereafter (see Fig. 2) (Svenningsen 2010).

Fig. 2.

Illustration of reported and simulated wind turbine power curve, and turbine power coefficient (C_p) (see Eqn 1). This example is for a GE Energy 2.5xl wind turbine model used at Mumbida wind farm.

The turbine power coefficient is a function of wind speed and the aggregate of the aerodynamic, mechanical and electrical efficiency. In addition, wind turbine self-load reduces net power output. For commercial wind turbines, the various aspects of efficiency and self-load are aggregated to give a wind power curve for the specific model of wind turbine. These curves are produced by wind turbine manufacturers according to the requirements of the International Electrotechnical Commission IEC 61400-12. Wind power is reported against wind speed at discrete (piece-wise) 0.5 m s⁻¹ wind speed intervals, normalised to the International Organization for Standardization (ISO) International Standard Atmosphere (ISA) density of 1.2250 kg m⁻³. A representative wind power curve is shown in Fig. 2.

For this study, the wind power curve was simulated as a continuous parametric function. The advantage of a parametric function over a piece-wise function is that the parameters can be readily adjusted over a continuous range for optimisation. A parametric function that approximately replicates the shape of wind power curves is the logistic function, of which various configurations are available (Villanueva and Feijóo 2018). The logistic function used in this work is a three-parameter type, which represents a trade-off between precision and complexity. The complete transfer function, taking into account cut-in and cut-out speeds, is given in Eqn 2.

(2)P(t)={ a . e 1 + e−b(U ( t )−c) ifU(t)≥cut-inspeedandU(t)<cut-outspeed0otherwise

where P(t) is the simulated wind power at time t, a is the rated power⁴ of the wind farm, where it is assumed that the wind farm behaves as a wind turbine, c is the wind speed scale (midpoint of the wind power curve), b is the shape of the curve and U(t) is the wind speed at time t given by the reanalysis data and calculated from the u and v wind components using the Pythagorean theorem.

Reported wind speeds at discrete heights must be modified to give estimated wind speed at the turbine hub height. For MERRA-2 (Gelaro et al. 2017) and ERA5 (Hersbach et al. 2020), the reported wind speed at 50 m (MERRA-2) and 100 m (ERA5) was modified using the log law equation (Archer and Jacobson 2003) given below.

(3)U(z)=Urln(z z0 )ln( zr z0 )

where U(z) is the adjusted wind speed, z₀ is the surface roughness length, z is the hub height and U_r is the wind speed at the reference height, z_r. The surface roughness length is influenced by the vegetation type, with grasslands, croplands and pastures having the shortest lengths, whereas forests have the longest (Hahmann et al. 2020). Most of Australia’s wind farms are situated on agricultural land, including previously cleared grazing or cropping areas, some with scattered patches of woodland. Wind farms located near the coast are sometimes located in scrubland areas. To maximise wind turbine performance, wind farms require large, open spaces. Given these conditions, the surface roughness length for all wind farms was set to 0.1 m, corresponding to vegetation types with a short roughness length.

For BARRA-R2, wind speeds are reported at a range of heights enabling direct vertical interpolation. Log-interpolation was selected given the log-law relationship between height and wind speed within the surface layer of the atmospheric boundary layer. The surface layer is limited to approximately the lowest 100 m of the boundary layer.

A source of uncertainty and variability is the operational availability of wind turbines. For large wind farms, it is usual for one or more wind turbines to be unavailable owing to scheduled maintenance or unscheduled outages. The power scaling factor, e, is a heuristic that scales the rated power. It was initially set to 0.95 and adjusted by visual inspection of the time series graphs for each site so that the typical maximum simulated power was set to equal the maximum observed power of the wind farm. In practice, e will vary on an hour-by-hour basis; however, no reliable data could be found that would enable hourly calibration.

A related factor is that transmission constraints may reduce maximum power flows below the combined nameplate power of the wind turbines, or output may be curtailed in response to electrical operator instructions. By visual inspection of time series graphs, the power scaling factor appears to be a good guide under typical operating conditions.

Parameters v₀ and b describe the scale and shape of the power curve, and were set by minimising the error between the continuous function and the piece-wise power curve for the specific wind turbine. The resulting parameters are given in Table A3.

Given that wind power curves are normalised to the ISO ‘Standard Atmosphere’, an adjustment was required for air density. The recommended normalisation process for wind turbines with active power control is given in IEC 61400-12 (8.1), and recommends adjusting the wind speed using Eqn 4 below, and using the adjusted wind speed to determine the normalised wind power:

(4)Ud=U(ρρ0)1/3

where U_d is the adjusted wind speed, U is the given wind speed, ρ is the air density for the sample, ρ₀ is the standard air density of 1.2250 kg m⁻³.

Air density was not given directly in the reanalysis data. Using available data within each of the reanalyses (see Fig. 3 for the parameters used for each reanalysis), air density ρ was calculated from barometric (total) pressure P_t, partial pressure of water vapour P_v, air temperature T and the gas constant for dry air R_d, for each hourly sample at the appropriate location, using the approximation given in Eqn 5. Partial pressure of water vapour P_v was calculated with Teten’s approximation, given in Eqn 6, where RH is relative humidity, T is temperature (°C) and P_v is pressure (kPa).

(5)ρ=Pt−PvRd×T

(6)Pv={RH×0.61078exp( 17.27T T+237.3 )T≥0°CRH×0.61078exp( 21.875T T+265.5 )T<0°C

Fig. 3.

Block diagram of the validation model used for this study. Wind velocity given as zonal (u) and meridional (v) components. Wind speed adjustment uses International Electrotechnical Commission (IEC) standard for wind turbine performance reporting. BARRA-R2 labels: ta50m, air temperature at 50 m, psl, air pressure at mean sea level, hurs, near-surface relative humidity. MERRA-2 labels: T10M, air temperature at 10 m; SLP, air pressure at sea level; QV10M, specific humidity at 10 m.

2.6. Wind turbine parameters

Table A3 lists the specifications for the wind turbine models used in this study. The class was determined according to the closest fit to the power law curves given in Engels et al. (2009) (see Fig. 4).

Fig. 4.

Wind turbine models with respect to blade diameter, power and wind class. Curves based on equations from Engels et al. (2009).

2.7. Validation

We followed the recommendation of Borsche et al. (2015), testing for Pearson correlation, root mean square error (RMSE) and bias. We introduced two additional tests – the error between the simulated and observed frequency distribution of power, and the similarity between the simulated and observed ramp rates. Overall, reanalyses were validated in the time domain (x-axis is time), power domain (x-axis is power) and energy domain (integral of power over time) (see Table 2 and Fig. 3). Validation was performed using five statistical parameters as follows:

1. Pearson correlation (R) to test for hourly correlation between observed and simulated power in the time domain. Pearson correlation is an indicator of the similarity between the temporal profile of simulated and observed generation, but not necessarily the magnitude. If the highest simulated results tend to match the highest observed results and vice versa, the correlation is high.

2. Bias is a measure of the capability of the simulations to reproduce the observed annual capacity factor. It is measured as the error between the simulated and observed wind power annual capacity factor. Graphical results are presented in two ways: as the natural log difference between simulated and observed so that under- and over-estimation is presented symmetrically, and as a percentage. As local land surface and topography is important, the capacity factor results are compared with the Global Wind Atlas (GWA 3.1), which produces capacity factor estimates at 250- × 250-m resolution.

3. RMSE, which is an indicator of the difference between observed and simulated power with respect to both the magnitude and the temporal profile. As the rated power varies across wind farms, the observed and simulated data are firstly non-dimensionalised by dividing by the rated power so that the wind farms are compared on a common basis. The resulting RMSE embeds the temporal correlation, as well as the additive and multiplicative bias relative to the wind farm capacity.

   RMSE is given in Eqn 7.

   (7)RMSE= 1n ∑ i = 1n ( S i– O i) 2
   

   where S_i is simulated, O_i is observed and n is sample size.

4. The capability of the simulations to reproduce the observed frequency distribution of power is measured by comparing the overlap of the respective probability density functions (PDFs), described below.

5. The similarity between observed and simulated power of adjacent time steps (1 h) is a measure of the hourly ramp rate. Similarity is expressed as the overlap of observed and simulated hourly change histograms, described in more detail below.

Table 2.Types of statistical assessment used for this study.

Domain	Statistical measure
Time	Pearson correlation
	RMSE (non-dimensionalised)
Power	Power frequency distribution error
	Overlap of the hourly change histogram
Energy	Annual capacity factor bias

As validation in the power domain is novel in the wind reanalysis literature, it is described as follows (validation of wind speed distribution has been undertaken in the climatology literature, e.g. Hahmann et al. 2020). Probability density functions of wind power were constructed for both observed and simulated wind power. The PDFs were compared piece-wise using the intersection (overlap) equation from Cha (2007), given in Eqn 8. The overlap (0–1) of the PDFs provides a simple estimate of the similarity of the shape of the respective power distributions, and therefore the validity of the simulated wind power in the power domain (see Fig. 5 and 6).

(8)Op=∑i=1nmin(Mi,Ni)

where O_p is overlap in the power domain, M_i is the frequency of values in a given bin for observed, N_i is the frequency of values in a given bin for simulated, n is number of bins.

Fig. 5.

Illustration of power frequency distribution overlap between the observed (NEM and SWIS) and simulated probability density functions (PDFs).

Fig. 6.

Illustration of change-of-power overlap (see Eqn 9).

In addition to the power distribution, we also test for the change of power, or equivalently, hourly ramp rates. This is measured by taking the power difference between adjacent time steps and sorting into histogram bins. The observed and simulated histograms are compared as the overlap of the respective bins, also using the equation from Cha (2007), given in Eqn 9. The result is expressed as the overlap (0–1), and provides an indication of the similarity of the respective histograms.

(9)Oc=∑i=1bmin(Zi,Wi)

where O_c is overlap of the change of power, Z_i is the frequency of values in a given bin for observed, W_i is the frequency of values in a given bin for simulated, b is number of bins.

3.Results

In this section, we present the results, beginning with the reanalysis wind speed distributions. Following this, a snapshot of 4-day time series results is presented, illustrating representative behaviour of the simulations against the observed power. Snapshots of wind droughts are also presented, illustrating the capability of the simulations to reproduce these. Subsequently, the correlation measures are presented, which demonstrate the capabilities and weaknesses of the simulations. Bias results are presented including statistical measures that support estimation of confidence intervals. The results from the power domain are presented, which point to the ways in which the simulations differ from the observed power.

3.1. Wind speed distribution

Wind energy researchers have shown that the Weibull distribution fits probability densities for wind speed quite well (Christofferson and Gillette 1987). The Weibull distribution can be described by a shape factor K and a scale factor C. In order to test whether the reanalyses were consistent with a Weibull distribution, each of the reanalyses was fitted on an annual basis using a Weibull fitting function. (The Python scipy.stats function was used, which minimises the negative log-likelihood function; see https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.weibull_min.html.)

Across reanalyses and years, the shape and scale factors are broadly consistent with a Weibull distribution of shape factor of 2.1–2.5. An illustrative series of plots, showing both the distribution of wind speed and accompanying Weibull distributions with best-fit parameters are shown in Fig. 7. These visually show a good fit of the data to the Weibull distribution. The estimated Weibull shape K and a scale C factors for BARRA-R2 are given in Table A4 in the Appendix. The shape and scale parameters for the three reanalyses are shown as box and whisker plots in Fig. A1a, b. For all plots, the box extends from the lower to upper quartile values of the data (the interquartile range, IQR), with a line at the median. The upper whisker extends to the last datum less than (Q3 + 1.5 × IQR), whereas the lower whisker extends to the first datum greater than (Q1 – 1.5 × IQR).

Fig. 7.

Wind speed distribution and best-fit Weibull parameters – illustrative example for Albany, 2021. Solid lines are wind speed data. Dashed lines are Weibull plots using accompanying best-fit parameters. Also shown are the cut-in and cut-out speeds for the Enercon E66-1800 wind turbine used at the Albany wind farm.

3.2. Time series results

Two illustrative time series plots are shown in Fig. 8a, b. These are 4-day snapshots with hourly resolution. The choice of a 4-day time interval is intended to provide a balance between a sufficiently long time horizon to display the movement of weather patterns, without overwhelming the plot with detail. The first depicts a snapshot during which there are clear weather patterns producing high and low wind speed, while also giving good correlation at an hourly interval. The second depicts another 4-day snapshot, which also replicates weather patterns, but which demonstrates much greater variability with respect to hourly correlation. In general, all reanalyses are good at reproducing the general trends of the passage of weather systems, but the reproduction at shorter time intervals is highly variable.

Fig. 8.

Two illustrative time series plots. Thick blue line is observed power as reported by electrical system operator. Dashed lines are for simulated power using BARRA-R2, MERRA-2 and ERA5 reanalyses. Correlation and capacity factor on right side is for the 4-day snapshot. (a) Collgar 2014, 4-day snapshot for hour of year 100 to 196. (b) Albany 2018, 4-day snapshot for hour of year 200 to 296.

An important result is that the reanalyses are very good at reproducing extended wind lulls. Although hourly variability remains, the simulations generally show low power for most of the duration of lulls. Two representative snapshots, one depicting a 180-h lull and the other a 120-h lull, are shown in Fig. 9a, b.

Fig. 9.

Illustrative wind lulls for Musselroe for 2014 and 2016. (a) 300-h snapshot showing a 180-h lull during 2014. (b) 300-h snapshot showing a 120-h lull during 2016.

3.3. Correlation

The correlation results are presented as box plots for Pearson correlation and RMSE after non-dimensionalisation (Fig. 10 and 11, Tables A5–A6). These show generally good correlation but a wide variance. Pearson correlations are also presented as scatter plots against ground elevation to test whether these parameters are predictive of correlation (see Fig. 12). However, ground elevation does not add useful information.

Fig. 10.

Pearson correlation between modelled and observed wind power, all wind farms, 2014–2021.

Fig. 11.

RMSE between modelled and observed wind power, non-dimensionalised, all wind farms, 2014–2021.

Fig. 12.

Pearson correlation v. ground elevation, all wind farms, 2014–2021. Each site–year reanalysis is one data point. The R² for the linear trend lines is no higher than 0.11, suggesting that ground elevation is not predictive of Pearson correlation.

At longer time intervals, temporal aggregation appears to improve simulation quality with respect to correlation. This is demonstrated by the average monthly capacity factor graphs. Two representative sites are shown in Fig. 13 and 14. These graphs demonstrate two aspects. Firstly, the correlation between simulated and observed is high after monthly averaging. Secondly, the bias for all simulations is consistent across the year (Table 3).

Fig. 13.

Average capacity factor by month, Collgar, 2018. Bias for each reanalysis, averaged over 1 month, is similar across 1 year.

Fig. 14.

Average capacity factor by month, Musselroe, 2018. Bias for each reanalysis, averaged over 1 month, is similar across 1 year.

Table 3.Summary of Pearson correlation between modelled and observed wind power, all wind farms, 2014–2021.

Quartile	BARRA-R2	MERRA-2	ERA5
Upper	0.86	0.79	0.85
Median	0.81	0.75	0.77
Lower	0.76	0.70	0.73

3.4. Bias

Bias is measured as the error between the simulated and observed wind power annual capacity factor. Error is plotted with box plots as the difference in natural logarithm in Fig. 15 so that over- and under-estimation bias is plotted symmetrically. Fig. 16 plots the same data on histograms, shown as percentage bias. Table 4 plots the data in tabular form. The downward bias of ERA5 is clear on both types of plots, whereas a slight downward bias is evident for BARRA-R2 and MERRA-2.

Fig. 15.

Bias between modelled and observed wind power, all wind farms, 2014–2021, shown as difference of natural logarithms. Dashed reference lines shown for simulation v. observed being double (y = 0.693), half (y = 0.693) and quarter (y = −1.386). Intermediate dashed lines for intermediate equal steps (+20, +40%, etc.).

Fig. 16.

Bias between modelled and observed wind power, all wind farms, 2014–2021, shown as percentage. Each data point is one site–year, plotted with 31 bins.

To evaluate whether bias could be attributed to log-law extrapolation (applicable to MERRA2 and ERA5), we performed a linear regression of bias against hub height across all wind farms. Results indicate a slight overestimation bias with increasing hub height for all reanalyses, including BARRA-R2, which employs direct interpolation, suggesting that log-law extrapolation was not a reliable predictor of bias. Further details are included in the Appendix.

In order to use these results to estimate confidence intervals, it would be useful to know if they are consistent with a normal distribution. To test for normality, we apply D’Agostino and Pearson’s test.⁵ The resulting P values are 0.21 (BARRA-R2), 0.32 (MERRA-2) and 0.01 (ERA5) (n = 221). From these, we can deduce that the ERA5 bias samples are likely not normally distributed, whereas BARRA-R2 and MERRA-2 are potentially normally distributed. For completeness, the resulting mean and standard deviation assuming normally distributed bias are given in Table 5.

Table 4.Bias between modelled and observed wind power, given as median and quartiles.

Quartile	BARRA-R2 (%)	MERRA-2 (%)	ERA5 (%)
Upper	+5	+7	−17
Median	−7	−7	−30
Lower	−25	−25	−45

In energy modelling, it is standard practice to adjust for bias across a region using operating wind farms for calibration. Assuming a hypothetical site with a simulated (BARRA-R2) capacity factor of 30%, applying bias correction with ±1 s.d. (giving 68% of normally distributed data samples) would give a bias corrected capacity factor range of 21.6–33.3%.⁶

Such a range is much wider than would normally be assumed in energy system models.

Table 5.Normal distribution statistics for bias between modelled and observed wind power.

Statistic	BARRA-R2 (%)	MERRA-2 (%)	ERA5 (%)
Mean	−8.4	−8.9	−28.6
Std	19.5	22.1	20.3

ERA5 not normally distributed. See text.

A summary of capacity factors is given in Table A7 in the Appendix, and a summary of bias across all sites and years in Table A8 and Fig. 13.

3.5. Power domain

The power distribution graphs illustrate one way in which the simulated results differ from the observed data. The main difference is the presence, or not, of a peak at high power. All of the simulated results present as a bimodal distribution, with a peak at both low and high power. However, many of the observed samples present as a unimodal distribution, with a peak at low power only. A sample of power distributions is presented in Fig. 17a, b.

Fig. 17.

Sample of power frequency distributions for two wind farms, 2018, full year. The second (right) peak is clearly evident for Musselroe (top figure), but is flattened for Albany (bottom figure). (a) Musselroe, 2018. (b) Albany, 2018.

The lower peak is explained by the Weibull-like wind speed distribution of the sampled sites, which exhibits a peak towards lower wind speeds. The higher peak is explained by the extended plateau of the wind power transfer function, which essentially ‘bunches up’ the power distribution over a wide range of wind speeds.

Associated with this is the tendency for the observed wind power to fluctuate towards, but rarely remain at, rated power. Fig. 18a, b illustrates this tendency. The 4-day snapshot presents a 2-day period during which all reanalyses reported high continuous wind conditions. The difference between observed and simulated is evident in Fig. 18b, which shows a distinct peak near rated power for BARRA-R2 and MERRA-2, and slightly less so for ERA5, but for the observed, the same peak is shifted downwards (lower power) and flattened.

Fig. 18.

Woolnorth 2021, 4-day snapshot for hours 600–696. (a) Time series graph showing all the reanalyses reported high wind conditions for hours 650–690, while the observed (NEM) power fluctuated towards rated power. (b) Power frequency distribution displaying the same information as the above graph, but in the power domain (x-axis is power). The full power peak is evident for the simulations. The observed peak is shifted downwards (lower power) and flattened. Graph is produced using kernel density estimation.

The difference between simulated and observed power distribution overall is shown is Fig. 19. The goodness-of-fit was estimated as the overlap of the simulated with the observed. A higher overlap indicates a better fit. The results are presented as a box plot, showing a wide variance, with similar results for BARRA-R2 and MERRA-2. ERA5 shows a wider range, extending to a lower fit.

Fig. 19.

Power frequency distribution overlap between modelled and observed wind power, all wind farms, 2014–2021.

Fig. 20 is a representative set of histograms for the hourly change in power for adjacent time steps. The goodness-of-fit was also estimated as the overlap of the simulated with the observed. The performance of all the reanalyses is generally good (see Fig. 21), with the performance of BARRA-R2 the highest, followed by MERRA-2, then ERA5. In general, the simulations tend to show a peakier and narrower distribution than observed.

Fig. 20.

Hourly change (1-h ramp rate) as proportion of rated power, shown with 41 bins. Illustrative sample for Albany, 2019. The x-axis is percentage change, y-axis is number of samples; 8759 data points for 1 year.

Fig. 21.

Histogram overlap for hourly change as proportion of rated power (see sample Fig. 20), simulated with observed, all wind farms, 2014–2021. A higher overlap indicates a better fit.

4.Discussion

The two main purposes for wind power modelling using reanalysis products are for: (1) capacity factor estimation, and (2) wind power trace production for use in energy system models.

For capacity factor estimation, the critical accuracy measures are bias and bias spread. In principle, given a narrow bias spread, bias can be readily corrected using linear wind speed scaling. However, there are two limitations with wind speed scaling. The first is that the bias spread of all the reanalyses is high, giving much wider confidence intervals than would normally be required for energy system models. Unless such uncertainty is accounted for in these models, the resulting techno-economic assessments may be unreliable. For example, a simulated capacity factor of 30% using BARRA-R2 that gives a levelised cost of electricity (LCOE) of A$73 MWh^–1 would translate to a range of LCOE of A$65–101 MWh^–1 for a bias corrected range with ±1 s.d. (LCOE assumptions: capital expenditure, CAPEX, A$1800 kW^–1; operational expenditure, OPEX, A$22.0 kW^–1 year^–1; 25-year project life, 8% discount rate).

The second limitation is that it is not clear what the physical basis of universal wind speed scaling is. In principle, scaling at a local level to account for topography or elevation, for example, may be justified. However, applying a universal scaling factor across an entire region would seem to be undermining the physical basis of reanalyses.

BARRA-R2 exhibited the narrowest bias spread, followed by MERRA-2, then ERA5. Both BARRA-R2 and MERRA-2 biased slightly under (median −7%), whereas ERA5 biased strongly under (median −30%). When used to provide seasonal or annual capacity factor estimates, consideration should be given to confidence intervals. The lower to upper quartiles for BARRA-R2 are −25 to +5%, for MERRA-2 are −25 to +7%, and for ERA5 are −45 to −17%. In relation to MERRA-2 and ERA5, the variance in bias from the present study is broadly consistent with results from regional studies for other regions in the world, e.g. Staffell and Pfenninger (2016); Jourdier (2020); Gruber et al. (2022).

Where possible, calibration with operating wind farms that are sited within the same wind regime (Gunn et al. 2023) should be undertaken to narrow confidence intervals. For estimation of bias confidence intervals, an assumption of normality for BARRA-R2 and MERRA-2 may be appropriate; however, ERA5 bias was found to be likely not normally distributed.

For wind power trace development, the performance at an hourly time interval is important. Critical hourly measures include correlation and rate of change. For all reanalyses, hourly correlation was highly variable. At times, there was substantial disagreement between simulated and observed wind power. Overall, BARRA-R2 exhibited the highest hourly Pearson correlation with median 0.81, followed by ERA5 at 0.76 and MERRA-2 at 0.75. These results are also reflected in RMSE, with BARRA-R2 showing the best performance. Although hourly performance is variable, the correlation for all reanalyses is very high at a monthly and seasonal time interval. Much of the noise that is apparent at short time intervals is ‘washed out’ at longer time intervals, consistent with the capability of the reanalyses to broadly replicate the passage of weather systems. This is consistent with other studies (e.g. Cannon et al. 2015; Staffell and Pfenninger 2016; Gruber et al. 2022) that found that temporal aggregation increased simulation quality.

Based on a preliminary data set, a hypothesis was that correlation was improved at those sites that are at low ground elevation. A cause that was considered was that reanalyses perform better at wind speed estimation in locations without significant topographical detail. In order to test for this, the dataset was tested for linear correlation between elevation and Pearson correlation (see Fig. 12). Although a slight correlation is visually evident on the plot, the R² for all reanalyses for elevation was no higher than 0.11, demonstrating almost no correlation. Other factors that may be relevant include the type of land cover, vegetation and surface roughness. A related hypothesis was that reanalysis performance may improve over time. However, for those sites with several years of data, no time-dependant trend was observed.

A factor that may have possibly contributed transient errors is the passage of weather fronts. Their passage is often associated with strong winds and a shift in wind direction, and they are known to be associated with sharp increases or decreases in wind power (Pichault et al. 2021). Simmonds et al. (2012) developed a method for identifying cold fronts in the Southern Hemisphere from reanalysis data. They observed that a shift of wind direction from the southwest to the northwest quadrant co-incident with an increase in the meridional wind (v) component greater than 2 m s⁻¹ was predictive of cold fronts. In order to investigate whether detection of frontal activity added information, the error was visually compared with wind direction changes based on 1-h up to 6-h lags. However, no discernible link was found.

Despite exhibiting variable correlation at short time intervals, all reanalyses generally capture wind droughts, defined here as region-wide continuous periods of at least 1 day of wind power output below 10% of capacity. Wind droughts that are co-incident with solar droughts represent the most critical periods from an operational and planning perspective. Owing to lower average insolation, the winter season is the most critical. The depth and duration of wind and solar energy droughts determine the capacity of dispatchable generation and/or electricity storage required to maintain system reliability (Gilmore et al. 2022). Prediction of wind droughts is therefore important from a system perspective. The confidence of wind drought simulation is high.

Analysis in the power domain revealed limitations in the use of reanalyses for wind power trace production. In order to explore these limitations, we identify two factors: (1) the wind speed distribution, and (2) the transfer function that governs the transformation of wind speed into wind turbine power. Wind speeds have been shown to be described by the Weibull distribution, which shows a peak at low wind speeds. The speed-to-power transfer function for wind turbines is typically described by a logistic function, with an extended plateau from the rated speed, up to the cut-out speed. This gives a peak near the rated power of the wind turbine. Given these two factors, the wind power distribution would be expected to exhibit two peaks – the first due to the wind speed peak at low wind speeds, and the second peak at higher power due to the extended plateau of wind power.

In all simulated results, both peaks in the frequency distribution of wind power were evident, although the magnitude of the higher peak varied across different sites. Notably, the higher peak appeared significantly less frequently than anticipated in the observed power distributions. Additionally, although the simulations often exhibited brief, continuous periods of rated power, it was rare for the observed power to sustain levels at or near rated capacity for extended durations.

This outcome is surprising, particularly in light of the wind speed-to-power transfer function. Notably, 12 out of 54 sites displayed a clearly defined higher peak, whereas the remaining sites did not. This indicates that, at high wind speeds, the modelled wind power does not accurately reflect the observed behaviour. Overall, the performance of BARRA-R2 and MERRA-2 was comparable, whereas ERA5 performed less effectively in this context.

Such behaviour may be related to factors that are important at the scale of a wind farm. One possibility may be that additional turbulence at high wind speeds reduces flow normal to the turbine blades. Associated with this is that high wind speeds are often associated with wind direction changes, requiring active turbine yaw changes. Under such dynamic conditions, wind turbines may be unable to maintain optimal yaw.

Wind turbines are optimised for a range of wind speeds, defined by the class. Approaching the cut-out speed, wind turbines are programmed to either feather to reduce the angle of attack, or to increase the angle of attack until stalling, both of which reduce lift (Burton et al. 2011). Rotor speed is typically regulated to provide a fixed tip speed to wind speed ratio. Although active control occurs in real time, dynamic wind conditions, especially changes in wind direction and turbulence, may reduce power output below that predicted based on steady-state conditions. Assuming a distribution of wind speeds across a wind farm, it is possible, during high average wind conditions as reported by hourly data, that some turbines are operating at or near their cut-out speed. A weakness of reanalyses generally is that wind speed is reported as grid cell averages for an entire geographic grid.

It is noteworthy that wind farm developers undertake detailed onsite measurement at anticipated wind turbine hub heights, often coupled with detailed flow analysis, to determine the optimum location of turbines. Such an approach suggests that variation in wind speeds at a local level is significant. The role of orography is important, owing to increases in wind speed on ridges or escarpments. Such ground surface information lies below the spatial resolution of reanalyses. Interestingly, the micro-scale spatial resolution of the GWA did not improve bias accuracy, and performed worse than uncorrected BARRA-R2 simulated results. In one study using Italy as a case study, Cavaiola et al. (2023) compared a high-resolution downscaled reanalysis based on ERA5 with a machine learning approach using coarser reanalysis wind speed, and found that the machine learning approach outperformed the downscaling approach.

The spatial distribution of wind flows across a wind farm can be partly addressed using computational fluid dynamics (CFD) tools, which enables more detailed estimation of the spatial dynamics of wind flows (Letzgus et al. 2022). However, such modelling is far more detailed and site-specific than practical for the purpose of high-level energy system modelling. Related to this is the role of wake effects, which reduce power below that expected by simulation for wind directions aligned with turbine layout. Wind turbine layout is governed by several factors, the most important of which is to maximise annual output while maintaining a minimum distance between turbines in the prevailing wind direction to prevent any excessive wake effects (Serrano Gonz´alez et al. 2014). As such, wake effects are a source of uncertainty in simulation modelling, and would be expected to lead to overestimation of annual output.

Related to changing wind conditions is the sub-hourly ramp rate. From an electrical system operational perspective, it is more critical to be able to predict ramp rates at shorter time intervals than 1 h. Additionally, ramp-downs are generally more important for system reliability than ramp-ups. Given the limitation of the 1-h time step of the reanalyses, sub-hourly ramp rates could not be determined. Nonetheless, the 1-h performance of all the reanalyses is generally good, with the performance of BARRA-R2 the highest, followed by MERRA-2, then ERA5. In general, the simulations tend to show a peakier and narrower distribution than observed.

One factor that was investigated but determined to be unimportant was the role of turbine inertia in damping the power rate of change. Large wind turbine rotors and blades possess significant inertia when rotating. Blade mass (kg) has been shown to approximate rad^2.30, where rad is swept radius (m), giving 8 tonnes for a 50-m swept radius (Veers et al. 2003). The inertia time constant determines the time interval during which a turbine can supply its rated power using only the kinetic energy stored in its rotational mass. The inertia time constant of wind turbines typically ranges between 2 and 6 s (Fernández-Guillamón et al. 2019). Given that this is two orders of magnitude shorter than the temporal resolution of the reanalyses, it was deemed insignificant.

Another factor that may influence bias is the difference between operational and prescribed performance. Wind turbine vendors report the measured power curve according to the requirements of IEC 61400. This is intended to be representative of all turbines of a specific model under steady-state conditions. However, the performance of individual turbines may differ from the prescribed performance owing to several factors. These may include the maintenance schedule, the climatic conditions to which turbines are subjected and age-related degradation. In one study, Staffell and Green (2014) assessed the UK wind farm fleet and found that turbines lose 1.6 ± 0.2% of their output per year. Age-related degradation was not explicitly accounted for in the present study, although the power scaling heuristic, e, should partly account for such degradation.

5.Conclusion

Regional atmospheric reanalyses are a convenient source of wind speed data for wind power modelling. Using desktop modelling tools, reanalyses enable energy researchers or commercial developers to produce capacity factor estimates, or wind power traces, for any location within the reanalysis region. Such traces can serve as important inputs into energy system models and techno-economic assessments. However, reanalyses are known to exhibit significant biases. Such biases and their uncertainty can significantly reduce the validity of energy assessments.

Bias has been widely investigated for the most popular global reanalyses, MERRA-2 and ERA5; however, the Australian-based reanalysis suites BARRA and the newer BARRA2 have not been validated against operating wind farms.

An aim of this study was to validate simulated wind power output using BARRA2 against observed output from operating wind farms – observed wind farm output represents the most reliable baseline in regards to energy system modelling and wind power estimates. Regional-level wind speed estimates can provide high-level estimates of wind power potential; however, many factors interact in complex ways at the scale of a wind farm. Investigation at the scale of a wind farm is one way of exploring and identifying those factors.

We found that all of the reanalyses broadly replicate wind speed patterns associated with the passage of weather systems, as well as replicating seasonal patterns. A notable strength is their capability of reproducing wind droughts. At short time intervals (i.e. 6-hourly, hourly), correlation was highly variable, with at times no correlation between simulated and observed power. Bias, measured as the error between observed and simulated capacity factor, was variable across sites.

Judged by five factors – bias, bias spread, hourly correlation, RMSE and the distribution of hourly change (hourly ramp rates) – BARRA-R2 gave the best results, followed by MERRA2, then ERA5. None of the reanalyses was able to consistently reproduce the observed power distribution of wind farms.

In order to translate bias and uncertainty into cost estimates, we estimated the LCOE for a hypothetical wind farm. Using BARRA-R2 with a simulated capacity factor of 30% and a LCOE of A$72 MWh^–1, translates to a cost range of A$65–101 MWh^–1 after bias correction with ±1 s.d.

At the level of a wind farm, it is apparent that site-specific factors are contributing to differences between simulated and observed power. These differences are evident in the frequency distribution of wind power and the hourly ramp rates, and are likely related to micro-scale wind behaviour and wind turbine response. It is also evident that some of the difference derives from wind speed error in the reanalyses. In considering their own study, as well as similar studies employing MERRA-2 and ERA5, Staffell and Pfenninger (2016) suggested that at least some of the problem lies in the underlying reanalysis wind speed data. The results presented here similarly suggest that reanalyses can capture wind speed at scales compatible with the reanalysis resolution, but are not able to replicate wind speeds at a spatial and temporal scale that is relevant at the level of a wind farm.

The relative success of machine learning in other studies demonstrates that it is possible to calibrate reanalyses with known data (Cavaiola et al. 2023), and points to the potential for improving simulation quality. A challenge remains as how to estimate, and narrow, the confidence intervals for those sites for which calibration data are not available.

Although reanalysis performance is good at times, we recommend the use of confidence intervals that reflect the degree of uncertainty in wind power simulation, and the degree of confidence required in the energy system model.