Streamlining the graphical forecast process

Keywords: Australian Bureau of Meteorology, BoM, forecast process, GFE, Graphical Forecast Editor, National Blend of Models, NWS, NWP, Operational Consensus Forecast, Regional Forecast Centres, streamlining, U.S. National Weather Service.

A challenge faced by meteorological agencies around the world is how to provide forecast services that make the best use of both the variety of increasingly skilful numerical weather prediction (NWP) models and the experience of their human forecasters, as the state of forecasting continues to evolve (Benjamin et al. 2018). During the 2000s, both the U.S. National Weather Service (NWS) and the Australian Bureau of Meteorology (BoM) adopted the Graphical Forecast Editor (GFE) software as their platform for more directly harnessing NWP outputs in a human-curated gridded forecast production process. From this common origin, the NWS and the BoM took somewhat different approaches to the implementation of the GFE in their forecasting operations.

Similar drivers operated to encourage both organisations to develop more streamlined approaches to graphical forecasting, namely the quest for greater forecast consistency and the desire to maximise the impact of forecasters in forecast service provision. In both cases these changes originated regionally rather than nationally. Development of streamlining in the U.S. has proceeded more at the grassroots level, whereas in Australia it has been more management driven. Independent developments in both countries have nonetheless arrived at solutions which are similar in many respects.

In this paper we will examine the streamlining of the graphical forecast process in more detail, comparing and contrasting the U.S. and Australian experiences, and noting some similar future directions anticipated by both organisations. Sturrock and Griffiths (2020) give a concrete example of the role of the forecaster within such a process.

During the 1990s, several new forecasting platforms were developed for the U.S. NWS (Glahn and Ruth 2003), one of which was the GFE. GFE’s purpose was to migrate decades of text-based forecasts into a graphical, gridded format. The gridded forecast consisted of graphical depictions for many weather elements such as temperature, wind, weather type, probability of precipitation and amounts of precipitation. When GFE was deployed to U.S. NWS offices in the early 2000s, to allow forecasters to create this new forecast, the application came with:

1. a set of algorithms known as smart initialisations to bring models into GFE;

2. graphical editing tools (‘smart tools’) and the ability to create your own tools;

3. natural language generation based on the gridded forecasts, to enable efficient production of a large number of text forecasts; and

4. ability to generate custom forecast graphics to post to the internet.

The GFE’s hierarchical framework for customisation allowed for a vast amount of local and regional development that continues to the present day. When grids from the 122 different Weather Forecast Offices were combined in a mosaic to produce national grids, the diverse set of tools per office used in different ways along with a variety of forecast models to populate from, resulted in discontinuities at the boundaries between offices. These inconsistencies were documented early in the deployment of GFE (Abrams 2004).

The BoM in Australia also desired to modernise its forecast production and chose to adopt the GFE software following its introduction in the U.S. NWS (Hart 2019). The system required significant modification to customise it for Australian conditions. The initial ambitious plan was to implement the GFE in the BoM’s seven Regional Forecast Centres within 2 years. Development was undertaken in a centralised fashion, partly to manage within tight timelines and limited resources. Implementation decisions were also guided by reflection on the U.S. experience. To avoid the proliferation of different tools and forecast processes that occurred in the U.S., the GFE in Australia was much less locally configurable. A national set of smart tools and text formatters was developed to promote more consistent forecasts between the Regional Forecast Centres. To reduce the number of different possible forecast starting points, only a small range of forecast guidance was provided.

The full implementation actually required an additional 6 years and supplementary funding from the Australian Government. The roll out to the final Regional Forecast Centre was completed in 2014.

Recognising the consistency issues, the U.S. NWS Central Region (one of four regions in the contiguous U.S., containing 38 Weather Forecast Offices) chartered a labour (union) management team in 2008 titled the Central Region Grid Methodology Advisory Team (CRGMAT). After performing an analysis of the current state of gridded forecasting operations across Central Region, the team composed a report containing eight recommendations to begin streamlining operations.

1. Retire the existing policy for the extended portion of the forecast (days 4 to 7¹), since it was not being followed at a significant number of offices.

2. Create the Smart Tool Advisory Task Force.

3. Create the Verification Task Force.

4. Investigate a new days 4 to 7 policy utilising a ‘model blend of the day’ approach, recognising the work of Mass and Baars (2005).

5. Continuation of the team well beyond its end date of September 2009.

6. Emphasise dedication to near-term grids.

7. Improvements to quantitative precipitation forecast (QPF) collaboration.

8. Formal investigation of post-processing to help ‘smooth out’ consistency issues.

The team tackled many of these recommendations in the decade that followed via a variety of projects. A timeline of projects that led to a more streamlined approach to gridded forecasting is shown in Table 1.

Table 1.  Timeline of implemented projects by the CRGMAT

Fig. 1.  Flow Diagram for ForecastBuilder. ‘Top-Down Grids’ reflect representations of the environment above the surface allowing ForecastBuilder to produce freezing rain and sleet (Robbins and Cortinas 2002). Italicised items can be initialised from the NBM.

The BoM has been using bias-corrected NWP model blend forecasts as an aid to producing point-based forecast services since the early 2000s (Woodcock and Engel 2005). From the start of the BoM’s move to gridded forecasting, there was a parallel effort to develop gridded model blend forecasts, to serve as a consistent starting point for gridded forecast production (Engel and Ebert 2012). These were known as Gridded Operational Consensus Forecasts or GOCF.

In the New South Wales (NSW) Regional Office, work began to develop a standardised forecast process which made use of the GOCF forecasts. The impetus for this came from local management in the NSW office, who were concerned about the additional workload arising from the move to grid-based forecasting over 7 days. To keep forecasters’ workload manageable, they promoted use of a standard process for the long-term forecast, while keeping the focus of forecaster effort on high-impact and short-term forecasting. A qualitative verification study of the standard process was conducted (Griffiths and Park 2012), comparing the performance of forecaster-produced grids for days 5 to 7 with performance of the grids and text forecasts which would have resulted from following the standard process. This provided confidence that following the standard process would not obviously degrade the quality of long-term forecast services for NSW. In 2012 the process was codified into a GFE application named ‘FirstCutForecast’ by local meteorologists. Forecasters began to run FirstCutForecast as part of standard operating procedures for the day 5 to 7 grids in the NSW Regional Forecast Centre, with forecaster intervention in these grids only if there was a strong justification based on community impact.

Although the FirstCutForecast was made available as a tool in the national Australian GFE code base, its usage was low in offices other than NSW. This standard approach did not receive the same backing from local management in the other offices. There were limitations with early versions of the tool. The range of weather types covered by FirstCutForecast was initially very restricted, with no thunderstorms, drizzle, fog or frost. All precipitating weather was labelled as ‘showers’ or ‘snow showers’ depending on snow level. Other weather types had to be put in by forecaster intervention in the grids. The reliance on GOCF guidance was also an obstacle. Forecasters had preconceptions regarding poor quality of the guidance and discomfort with blended guidance, in which meteorological features were smeared out depending on the degree of model disagreement. There was concern at the potential for forecast ‘flip flops’ if the forecast was being refreshed at each issuance with new guidance.

In 2015, work in the BoM’s research area was initiated to perform systematic verification of GFE grids and thus provide evidence for systematic improvements. This work merged with a broader organisational thrust to streamline the forecast production process. This aimed to give forecasters more time to devote to high-value activities such as providing decision support to weather sensitive sectors of the Australian community, while achieving greater forecast consistency through common processes.

Verification was used to explore the quality of the official forecasts produced from the GFE to see where the forecaster was adding value relative to the forecast guidance, taking the output of FirstCutForecast as a benchmark. The verification methodologies used were as presented by Griffiths et al. (2017) for precipitation but extended to a wider range of elements. The picture that emerged was that the GOCF-based FirstCutForecast outputs were by and large of comparable quality to the officially produced forecasts. It was also shown through use of a ‘flip-flop index’ (Griffiths et al. 2019) that forecasts based on consensus guidance were, in many cases, more stable than the official forecasts.

Concurrent to the verification work and guided by its results and by qualitative feedback from forecasters, there have been steady efforts to improve the quality of the GOCF guidance and improve the forecast process used in FirstCutForecast. The application has been extended to include more weather types, including thunderstorms and frost, as well as marine and fire weather grids. Fig. 2 shows the methodology employed in the current version of the FirstCutForecast application in GFE.

Fig. 2.  Flow diagram for FirstCutForecast. Italicised text shows places GOCF is used.

Verification has provided forecasters a better understanding of the quality of the GOCF guidance and outputs of the FirstCutForecast application. Verification has also guided development of national standard operating procedures. Introduced progressively in mid-2018 and mid-2019, the standard operating procedures include use of FirstCutForecast as the standard forecast process for the day 3 to 7 forecast. The procedures call for limited forecaster interventions where FirstCutForecast has known deficiencies, in cases where the weather would have significant impact on the community. Forecasters note times when they intervene in the standard process and record their rationale. The aim of this is to feed back to improvements in the standard operating procedures and the forecast guidance. Fig. 3 shows how national use of FirstCutForecast has increased across the seven Regional Forecast Centres during 2018 and 2019, reaching a similar level of usage to the NSW Regional Forecast Centre by the end of the period. Usage was measured by recording the number of times that forecasters ran the FirstCutForecast application. Note that this is only a coarse measure as it does not indicate how many of the forecast days were edited by the application on any particular run.

Fig. 3.  Number of daily usages of FirstCutForecast, averaged by month, in the NSW Regional Forecast Centre and averaged across all seven Australian Regional Forecast Centres during 2018 and 2019, extracted from GFE system logs.

Working with any software for several years leads to habits. Forecasters develop their own distinct ways of creating a gridded forecast, which was evident in the results from the 2009 CRGMAT report. However, several leadership qualities and best practices associated with this team helped move forecasting culture in the NWS’s Central Region to a more streamlined forecast process.

1. Team membership. The team consisted of field representatives from both management and union.

2. Communication with action. Forecasters could submit feedback to the team either directly via email or through surveys on the various projects being tested or already in place. The team would always respond to the feedback and incorporate as many of the suggestions as it could into smart tools, provided they fit the vision of the project. Additionally the team would hold webinars with the offices on occasion to provide them with updates.

3. Research-to-operations approach. The team would plan out the concept, get approval by the regional labour council (composed of management and union), test it in groups of offices and get feedback, adjust as necessary, obtain approval again by the regional labour council, and then deploy region-wide.

4. Trust. After the team successfully implemented the day 4 to 7 process, the team gained a lot of trust with other regional Weather Forecast Offices. Forecasters then were more willing to accept any future changes proposed by the team.

The Australian move towards a more streamlined forecast process was similar in that it originated in one part of the country. However, the Australian developments have been more top-down, with the initial impetus coming from local management rather than forecasters. There have been mixed consequences arising from the more top-down approach. With less contribution to the development, the forecasting staff has been less engaged with the changes. The industrial landscape in Australia is different to that in the U.S, with staff unions participating less in the detail of day-to-day work duties, and so unions have not been involved in the specifics of the graphical forecast process. With national management also supporting more streamlined forecast production, Australia has been able to achieve nation-wide adoption of what has been called the ‘Optimised Forecast Process’ using the common FirstCutForecast starting point, thus capturing more of the benefits of temporal and spatial consistency across the national forecast grids than has yet been achieved in the U.S.

Despite these positive changes, portions of the U.S. and Australian forecaster populations continue to struggle in this new forecast process paradigm. Lingering challenges include the following.

1. Loss of manipulating a gridded forecast/feeling ‘automated out of a job’. Giving up modifying the forecast feels like giving up what they dreamed of doing when they got into the field of weather. However, with growing decision support needs that expand beyond the gridded forecast, the need for the forecaster is higher than ever.

2. Concerns over how consensus forecasts handle extremes. Since blended model data sets can average out the extremes, forecasters worry that they cannot pick up on high-impact events. A common statement is, ‘Yes, it does well over the long haul but what about for events that really mean something?’ Given the nature of consensus forecasts, if there is good agreement on a high-impact event, the blend will reflect that.

3. Concerns over how weather scenarios are presented. Some forecasters prefer presenting a gridded forecast which represents a particular detailed scenario, with associated physical conceptual models, rather than using blended guidance where details are blurred by uncertainty.

4. Concerns over ‘what is the observation’ utilised for verification. To verify any science a source for ‘ground truth’ is required. The NBM in the U.S. utilises the Un-Restricted Mesoscale Analysis (URMA) as its truth and its source for bias correction. URMA merges together many different observation networks with the background High-Resolution Rapid Refresh model. Some forecasters take issue with this system as it tends to dampen the observations taken at Automated Surface Observing Systems (ASOS). However, many of these ASOSs are at airports where fewer people live and are more open, exposing them to fewer friction effects. In Australia, the Mesoscale Surface Analysis Scheme (Glowacki et al. 2012) is the source of bias correction for OCF. Forecasters in Australia had similar concerns about its suitability as a verifying analysis, so that in Australia the GFE verification has been performed against observations at sites.

Solving these challenges requires continued leadership by promoting the benefits of a streamlined forecast process consisting of a common starting point and a standardised grid production flow, and the advantages of channelling forecaster energy from routine forecast production into helping the community understand and respond to weather impacts.

As forecasters in the U.S. NWS and the BoM continue to adapt to a blend of human and machine interaction when it comes to gridded forecasts, there are several new areas to be explored.

1. Enhancing the first 24 to 36 hours of the forecast. Forecasters will continue to monitor and intervene in the short-term forecast, particularly to achieve consistency of public messaging of significant and impactful weather events within warning timeframes. Both the BoM and the U.S. NWS Central Region are planning initiatives to help keep the first few hours of the forecast current, making use of information from nowcasting products.

2. Probabilistic Forecast Services. Several projects involving probabilistic services have started in the U.S. NWS such as the Probabilistic Winter Precipitation Forecast and Probabilistic Quantitative Precipitation Forecast. The National Blend of Models is becoming more probabilistic as well. How these are created and communicated to our partners to enhance our routine gridded forecast will become increasingly important in the future. The BoM also produces probabilistic precipitation forecasts and is partnering with the UK Met Office in developing probabilistic guidance post-processing software which will enable more probabilistic forecast services in future. Better probabilistic guidance along with the ability of forecasters to properly message the uncertainty surrounding impactful weather events will undoubtedly shift the paradigm from the dependence on single-valued forecasts to those which are probabilistic in nature.

3. National Forecast Production. The U.S. WPC is already producing some gridded forecasts at 5-km resolution across the contiguous United States. The BoM is currently developing the capacity to run the GFE on a national Australian domain, initially at 6-km and ultimately 3-km resolution. The intention in Australia by the end of 2021 is to transition from using regional GFEs to a national GFE instance for forecast production based on an increasingly streamlined Optimised Forecast Process.

Both the Central Region of the U.S. NWS and the Australian BoM have been striving for a streamlined forecast process for several years since the adoption of GFE. It was evident early on in this process that having offices develop their own tools and generate the forecasts in different ways was very inefficient. Although the paths to reaching a streamlined forecast process were different, both agencies have arrived at similar solutions, creating blended model starting points and a common tool set. Nonetheless, both agencies continue to experience some challenges, which is to be expected as it takes time to change culture. However, continuous leadership pushing the benefits of a streamlined process will help office culture to evolve. As forecasters continue to adopt this new forecast preparation paradigm, more useful forecaster time can then be spent on (1) decision support services, (2) adding more details concerning impactful weather in the first 24 to 36 hour forecast period and (3) exploring new ways to incorporate and message probabilistic information in decision support.

The GFE has facilitated this incremental evolution of the forecast process to better harness the power of post-processed numerical weather prediction guidance and the insights of forecasters. This stands as a testament to the power of the original vision behind the GFE as well as to the wisdom of the strategic decision taken by both the U.S. NWS and the Australian BoM in the early- to mid-2000s to go down the route of graphical forecast production.

The scientific results and conclusions, as well as any view or opinions expressed herein, are those of the authors and do not necessarily reflect the views of NWS, National Oceanic and Atmospheric Administration, or the U.S. Department of Commerce.

The authors declare that they have no conflicts of interest.