A Practical Guide for Working with Weather Datasets

Report

This report categorizes weather datasets into 8 main types, of which 5 are historical data and 3 are forecast data. Each of the 8 dataset types is described, and an illustrative example of each is provided.

Report
Demo Computer Programs

Many weather datasets exceed 100 gigabytes and some are much larger. While most climate scientists have access to servers that can store and process massive weather datasets, other types of researchers may wish to perform weather analyses on a standard personal computer. A personal computer rarely offers more than 1000 gigabytes of storage space and 16 to 32 gigabytes of RAM (active memory for running applications and programs). Given these constraints, a researcher will need a clever approach for working with large weather datasets. This paper describes techniques for running analyses of a large weather dataset despite the storage and memory limitations imposed by a personal computer. This paper is accompanied by 6 small computer programs written in VBA, Python, R, and C++, zipped together into a single file to simplify the download process. These programs illustrate techniques for processing large weather datasets using the limited memory (RAM) available on a personal computer.

The Global Historical Climatology Network Daily (GHCNd) dataset consists of daily temperature, precipitation and wind speed observations collected from over 100,000 land-based weather stations. GHCNd provides geographic coverage of much of the world, but the availability of data varies significantly from one country to another, as well as within each country. In general, the density of weather stations is greatest in urban areas and lowest in rural areas.

To address the irregular spacing of ground-based weather stations, researchers have developed algorithms to translate station data into data on a regularly spaced geographic grid. The Climate Prediction Center – a federal agency that is part of the National Oceanic and Atmospheric Administration – uses station data to produce the Global Unified Temperature Dataset and the Global Unified Precipitation Dataset. These datasets cover the period from 1979 to the present and are gridded at 0.5 degrees (i.e. data is available every 0.5 degrees of latitude and longitude). For each day and for each grid point, the data provides minimum temperature, maximum temperature, and total precipitation. Note that the data does not include areas over oceans; rather, the data covers land areas.

“ERA5” is short for “Fifth generation European Center for Medium-Range Weather Forecasts Atmospheric Reanalysis of the Global Climate”. ERA5 is a “reanalysis” dataset. This type of data is produced by weather models that reconstruct the past in a manner consistent with available historical data, and consistent with the physical laws governing the earth’s atmosphere and oceans. The historical data fed into the model(s) can come from many sources, including weather stations, ships, planes, weather balloons, satellites, and radar. In contrast to station data which has an uneven distribution across space and time, reanalysis data is spatially and temporally complete. Reanalysis data is particularly helpful when analyzing historical trends in a region that has a low density of weather stations. ERA5 data extends from 1950 to the present in hourly time steps and provides coverage of the entire earth using a 0.25-degree geographic grid (i.e. data points are available every 0.25 degrees of latitude and longitude). The ERA5 dataset is updated daily, with a latency of only five days, and provides a comprehensive set of weather metrics including air temperature, air pressure, sea surface temperature, wind speed, total precipitation, and various measures of soil moisture.

IBTrACS stands for “International Best Track Archive for Climate Stewardship”. This dataset contains worldwide data on the paths and intensities of tropical cyclones that occurred between 1842 and the present.

The Storm Events Database, compiled by the National Oceanic and Atmospheric Administration (NOAA), contains data for severe weather events in the United States having sufficient intensity to cause loss of life, injuries, significant property damage, and/or disruption to commerce. This database covers the period from 1950 to the present, and provides loss estimates in addition to data describing each severe weather event.

A climate projection provides a sense of how the distribution of weather events may gradually change in the decades ahead. Perhaps the most well-known climate projections are those that feed into the climate assessment reports issued by the Intergovernmental Panel on Climate Change (IPCC). The IPCC issued its first climate assessment report in 1990. Since that time, an updated report has been released roughly every 5 years. The most recent report – the sixth in the series – was published in 2021. Because the modeling of climate change is complex, and because different models produce different results, the IPCC uses an “ensemble” of models to produce projections. For the latest IPCC report, the ensemble incudes over 100 models from more than 50 modeling centers. This ensemble is referred to as “CMIP6” which stands for “Coupled Model Intercomparison Project, Phase 6” (corresponding to the sixth report in the series). By running standardized greenhouse gas scenarios through each model in the ensemble, a range of forecasts is produced. The average results across all models serve as a best estimate, while the distribution of results provides some sense of the level of uncertainty.