Acosta, Mario C., Sergi Palomas, Stella V Paronuzzi Ticco, Gladys Utrera, Joachim Biercamp, Pierre-Antoine Bretonniere, Reinhard Budich, Miguel Castrillo, Arnaud Caubel, Francisco J Doblas-Reyes, Italo Epicoco, Uwe Fladrich, Sylvie Joussaume, Alok Kumar Gupta, Bryan N Lawrence, Philippe Le Sager, Grenville Lister, Marie-Pierre Moine, Jean-Christophe Rioual, Sophie Valcke, Niki Zadeh, and V Balaji, April 2024: The computational and energy cost of simulation and storage for climate science: lessons from CMIP6. Geoscientific Model Development, 17(8), DOI:10.5194/gmd-17-3081-20243081–3098. Abstract
The Coupled Model Intercomparison Project (CMIP) is one of the biggest international efforts aimed at better understanding the past, present, and future of climate changes in a multi-model context. A total of 21 model intercomparison projects (MIPs) were endorsed in its sixth phase (CMIP6), which included 190 different experiments that were used to simulate 40 000 years and produced around 40 PB of data in total. This paper presents the main findings obtained from the CPMIP (the Computational Performance Model Intercomparison Project), a collection of a common set of metrics, specifically designed for assessing climate model performance. These metrics were exclusively collected from the production runs of experiments used in CMIP6 and primarily from institutions within the IS-ENES3 consortium. The document presents the full set of CPMIP metrics per institution and experiment, including a detailed analysis and discussion of each of the measurements. During the analysis, we found a positive correlation between the core hours needed, the complexity of the models, and the resolution used. Likewise, we show that between 5 %–15 % of the execution cost is spent in the coupling between independent components, and it only gets worse by increasing the number of resources. From the data, it is clear that queue times have a great impact on the actual speed achieved and have a huge variability across different institutions, ranging from none to up to 78 % execution overhead. Furthermore, our evaluation shows that the estimated carbon footprint of running such big simulations within the IS-ENES3 consortium is 1692 t of CO2 equivalent.
As a result of the collection, we contribute to the creation of a comprehensive database for future community reference, establishing a benchmark for evaluation and facilitating the multi-model, multi-platform comparisons crucial for understanding climate modelling performance. Given the diverse range of applications, configurations, and hardware utilised, further work is required for the standardisation and formulation of general rules. The paper concludes with recommendations for future exercises aimed at addressing the encountered challenges which will facilitate more collections of a similar nature.
We present a variable-resolution global chemistry-climate model (AM4VR) developed at NOAA's Geophysical Fluid Dynamics Laboratory (GFDL) for research at the nexus of US climate and air quality extremes. AM4VR has a horizontal resolution of 13 km over the US, allowing it to resolve urban-to-rural chemical regimes, mesoscale convective systems, and land-surface heterogeneity. With the resolution gradually reducing to 100 km over the Indian Ocean, we achieve multi-decadal simulations driven by observed sea surface temperatures at 50% of the computational cost for a 25-km uniform-resolution grid. In contrast with GFDL's AM4.1 contributing to the sixth Coupled Model Intercomparison Project at 100 km resolution, AM4VR features much improved US climate mean patterns and variability. In particular, AM4VR shows improved representation of: precipitation seasonal-to-diurnal cycles and extremes, notably reducing the central US dry-and-warm bias; western US snowpack and summer drought, with implications for wildfires; and the North American monsoon, affecting dust storms. AM4VR exhibits excellent representation of winter precipitation, summer drought, and air pollution meteorology in California with complex terrain, enabling skillful prediction of both extreme summer ozone pollution and winter haze events in the Central Valley. AM4VR also provides vast improvements in the process-level representations of biogenic volatile organic compound emissions, interactive dust emissions from land, and removal of air pollutants by terrestrial ecosystems. We highlight the value of increased model resolution in representing climate–air quality interactions through land-biosphere feedbacks. AM4VR offers a novel opportunity to study global dimensions to US air quality, especially the role of Earth system feedbacks in a changing climate.
We describe the baseline model configuration and simulation characteristics of the Geophysical Fluid Dynamics Laboratory (GFDL)'s Land Model version 4.1 (LM4.1), which builds on component and coupled model developments over 2013–2019 for the coupled carbon-chemistry-climate Earth System Model Version 4.1 (ESM4.1) simulation as part of the sixth phase of the Coupled Model Intercomparison Project. Analysis of ESM4.1/LM4.1 is focused on biophysical and biogeochemical processes and interactions with climate. Key features include advanced vegetation dynamics and multi-layer canopy energy and moisture exchanges, daily fire, land use representation, and dynamic atmospheric dust coupling. We compare LM4.1 performance in the GFDL Earth System Model (ESM) configuration ESM4.1 to the previous generation component LM3.0 in the ESM2G configuration. ESM4.1/LM4.1 provides significant improvement in the treatment of ecological processes from GFDL's previous generation models. However, ESM4.1/LM4.1 likely overestimates the influence of land use and land cover change on vegetation characteristics, particularly on pasturelands, as it overestimates the competitiveness of grasses versus trees in the tropics, and as a result, underestimates present-day biomass and carbon uptake in comparison to observations.
We present the development and evaluation of MOM6-COBALT-NWA12 version 1.0, a 1/12∘ model of ocean dynamics and biogeochemistry in the northwest Atlantic Ocean. This model is built using the new regional capabilities in the MOM6 ocean model and is coupled with the Carbon, Ocean Biogeochemistry and Lower Trophics (COBALT) biogeochemical model and Sea Ice Simulator version-2 (SIS2) sea ice model. Our goal was to develop a model to provide information to support living-marine-resource applications across management time horizons from seasons to decades. To do this, we struck a balance between a broad, coastwide domain to simulate basin-scale variability and capture cross-boundary issues expected under climate change; a high enough spatial resolution to accurately simulate features like the Gulf Stream separation and advection of water masses through finer-scale coastal features; and the computational economy required to run the long simulations of multiple ensemble members that are needed to quantify prediction uncertainties and produce actionable information. We assess whether MOM6-COBALT-NWA12 is capable of supporting the intended applications by evaluating the model with three categories of metrics: basin-wide indicators of the model's performance, indicators of coastal ecosystem variability and the regional ocean features that drive it, and model run times and computational efficiency. Overall, both the basin-wide and the regional ecosystem-relevant indicators are simulated well by the model. Where notable model biases and errors are present in both types of indicator, they are mainly consistent with the challenges of accurately simulating the Gulf Stream separation, path, and variability: for example, the coastal ocean and shelf north of Cape Hatteras are too warm and salty and have minor biogeochemical biases. During model development, we identified a few model parameters that exerted a notable influence on the model solution, including the horizontal viscosity, mixed-layer restratification, and tidal self-attraction and loading, which we discuss briefly. The computational performance of the model is adequate to support running numerous long simulations, even with the inclusion of coupled biogeochemistry with 40 additional tracers. Overall, these results show that this first version of a regional MOM6 model for the northwest Atlantic Ocean is capable of efficiently and accurately simulating historical basin-wide and regional mean conditions and variability, laying the groundwork for future studies to analyze this variability in detail, develop and improve parameterizations and model components to better capture local ocean features, and develop predictions and projections of future conditions to support living-marine-resource applications across timescales.
A review of the results for GFDL models from the Computational Performance Model Intercomparison Project (CPMIP) raises the question of why the coupling costs appear so extreme, especially in light of tests demonstrating minimal cost for the FMS coupling framework. This technical memorandum seeks to explain the seemingly high cost associated with the FMS coupler in pre-industrial control (piControl) simulations using the GFDL ESM4 Earth system model and two configurations of the CM4 climate model1. While this technical note presents results for three specific piControl simulations, the findings are assumed to be general and applicable to the full range of configurations and experiments.
We describe the baseline coupled model configuration and simulation characteristics of GFDL's Earth System Model Version 4.1 (ESM4.1), which builds on component and coupled model developments at GFDL over 2013–2018 for coupled carbon‐chemistry‐climate simulation contributing to the sixth phase of the Coupled Model Intercomparison Project. In contrast with GFDL's CM4.0 development effort that focuses on ocean resolution for physical climate, ESM4.1 focuses on comprehensiveness of Earth system interactions. ESM4.1 features doubled horizontal resolution of both atmosphere (2° to 1°) and ocean (1° to 0.5°) relative to GFDL's previous‐generation coupled ESM2‐carbon and CM3‐chemistry models. ESM4.1 brings together key representational advances in CM4.0 dynamics and physics along with those in aerosols and their precursor emissions, land ecosystem vegetation and canopy competition, and multiday fire; ocean ecological and biogeochemical interactions, comprehensive land‐atmosphere‐ocean cycling of CO2, dust and iron, and interactive ocean‐atmosphere nitrogen cycling are described in detail across this volume of JAMES and presented here in terms of the overall coupling and resulting fidelity. ESM4.1 provides much improved fidelity in CO2 and chemistry over ESM2 and CM3, captures most of CM4.0's baseline simulations characteristics, and notably improves on CM4.0 in (1) Southern Ocean mode and intermediate water ventilation, (2) Southern Ocean aerosols, and (3) reduced spurious ocean heat uptake. ESM4.1 has reduced transient and equilibrium climate sensitivity compared to CM4.0. Fidelity concerns include (1) moderate degradation in sea surface temperature biases, (2) degradation in aerosols in some regions, and (3) strong centennial scale climate modulation by Southern Ocean convection.
Simulation of coupled carbon‐climate requires representation of ocean carbon cycling, but the computational burden of simulating the dozens of prognostic tracers in state‐of‐the‐art biogeochemistry ecosystem models can be prohibitive. We describe a six‐tracer biogeochemistry module of steady‐state phytoplankton and zooplankton dynamics in Biogeochemistry with Light, Iron, Nutrients and Gas (BLING version 2) with particular emphasis on enhancements relative to the previous version and evaluate its implementation in Geophysical Fluid Dynamics Laboratory's (GFDL) fourth‐generation climate model (CM4.0) with ¼° ocean. Major geographical and vertical patterns in chlorophyll, phosphorus, alkalinity, inorganic and organic carbon, and oxygen are well represented. Major biases in BLINGv2 include overly intensified production in high‐productivity regions at the expense of productivity in the oligotrophic oceans, overly zonal structure in tropical phosphorus, and intensified hypoxia in the eastern ocean basins as is typical in climate models. Overall, while BLINGv2 structural limitations prevent sophisticated application to plankton physiology, ecology, or biodiversity, its ability to represent major organic, inorganic, and solubility pumps makes it suitable for many coupled carbon‐climate and biogeochemistry studies including eddy interactions in the ocean interior. We further overview the biogeochemistry and circulation mechanisms that shape carbon uptake over the historical period. As an initial analysis of model historical and idealized response, we show that CM4.0 takes up slightly more anthropogenic carbon than previous models in part due to enhanced ventilation in the absence of an eddy parameterization. The CM4.0 biogeochemistry response to CO2 doubling highlights a mix of large declines and moderate increases consistent with previous models.
The imprint of anthropogenic activities on the marine nitrogen (N) cycle remains challenging to represent in global models, in part because of uncertainties regarding the source of marine N to the atmosphere. While N inputs of terrestrial origin present a truly external perturbation, a significant fraction of N deposition over the ocean arises from oceanic ammonia (NH3) outgassing that is subsequently deposited in other ocean regions. Here, we describe advances in the Geophysical Fluid Dynamics Laboratory's (GFDL) Earth System Model 4 (ESM4.1) aimed at improving the representation of the exchange of N between atmosphere and ocean and its response to changes in ocean acidity and N deposition. We find that the simulated present‐day NH3 outgassing (3.1 TgN yr−1) is 7% lower than under preindustrial conditions, which reflects the compensating effects of increasing CO2 (−16%) and N enrichment of ocean waters (+9%). This change is spatially heterogeneous, with decreases in the open ocean (−13%) and increases in coastal regions (+15%) dominated by coastal N enrichment. The ocean outgassing of ammonia is shown to increase the transport of N from N‐rich to N‐poor ocean regions, where carbon export at 100 m increases by 0.5%. The implications of the strong response of NH3 ocean outgassing to CO2 for the budget of NH3 in the remote marine atmosphere and its imprint in ice cores are discussed.
This contribution describes the ocean biogeochemical component of the Geophysical Fluid Dynamics Laboratory's Earth System Model 4.1 (GFDL‐ESM4.1), assesses GFDL‐ESM4.1's capacity to capture observed ocean biogeochemical patterns, and documents its response to increasing atmospheric CO2. Notable differences relative to the previous generation of GFDL ESM's include enhanced resolution of plankton food web dynamics, refined particle remineralization, and a larger number of exchanges of nutrients across Earth system components. During model spin‐up, the carbon drift rapidly fell below the 10 Pg C per century equilibration criterion established by the Coupled Climate‐Carbon Cycle Model Intercomparison Project (C4MIP). Simulations robustly captured large‐scale observed nutrient distributions, plankton dynamics, and characteristics of the biological pump. The model overexpressed phosphate limitation and open ocean hypoxia in some areas but still yielded realistic surface and deep carbon system properties, including cumulative carbon uptake since preindustrial times and over the last decades that is consistent with observation‐based estimates. The model's response to the direct and radiative effects of a 200% atmospheric CO2 increase from preindustrial conditions (i.e., years 101–120 of a 1% CO2 yr−1 simulation) included (a) a weakened, shoaling organic carbon pump leading to a 38% reduction in the sinking flux at 2,000 m; (b) a two‐thirds reduction in the calcium carbonate pump that nonetheless generated only weak calcite compensation on century time‐scales; and, in contrast to previous GFDL ESMs, (c) a moderate reduction in global net primary production that was amplified at higher trophic levels. We conclude with a discussion of model limitations and priority developments.
We document the configuration and emergent simulation features from the Geophysical Fluid Dynamics Laboratory (GFDL) OM4.0 ocean/sea‐ice model. OM4 serves as the ocean/sea‐ice component for the GFDL climate and Earth system models. It is also used for climate science research and is contributing to the Coupled Model Intercomparison Project version 6 Ocean Model Intercomparison Project (CMIP6/OMIP). The ocean component of OM4 uses version 6 of the Modular Ocean Model (MOM6) and the sea‐ice component uses version 2 of the Sea Ice Simulator (SIS2), which have identical horizontal grid layouts (Arakawa C‐grid). We follow the Coordinated Ocean‐sea ice Reference Experiments (CORE) protocol to assess simulation quality across a broad suite of climate relevant features. We present results from two versions differing by horizontal grid spacing and physical parameterizations: OM4p5 has nominal 0.5° spacing and includes mesoscale eddy parameterizations and OM4p25 has nominal 0.25° spacing with no mesoscale eddy parameterization.
MOM6 makes use of a vertical Lagrangian‐remap algorithm that enables general vertical coordinates. We show that use of a hybrid depth‐isopycnal coordinate reduces the mid‐depth ocean warming drift commonly found in pure z* vertical coordinate ocean models. To test the need for the mesoscale eddy parameterization used in OM4p5, we examine the results from a simulation that removes the eddy parameterization. The water mass structure and model drift are physically degraded relative to OM4p5, thus supporting the key role for a mesoscale closure at this resolution.
We describe GFDL's CM4.0 physical climate model, with emphasis on those aspects that may be of particular importance to users of this model and its simulations. The model is built with the AM4.0/LM4.0 atmosphere/land model and OM4.0 ocean model. Topics include the rationale for key choices made in the model formulation, the stability as well as drift of the pre‐industrial control simulation, and comparison of key aspects of the historical simulations with observations from recent decades. Notable achievements include the relatively small biases in seasonal spatial patterns of top‐of‐atmosphere fluxes, surface temperature, and precipitation; reduced double Intertropical Convergence Zone bias; dramatically improved representation of ocean boundary currents; a high quality simulation of climatological Arctic sea ice extent and its recent decline; and excellent simulation of the El Niño‐Southern Oscillation spectrum and structure. Areas of concern include inadequate deep convection in the Nordic Seas; an inaccurate Antarctic sea ice simulation; precipitation and wind composites still affected by the equatorial cold tongue bias; muted variability in the Atlantic Meridional Overturning Circulation; strong 100 year quasi‐periodicity in Southern Ocean ventilation; and a lack of historical warming before 1990 and too rapid warming thereafter due to high climate sensitivity and strong aerosol forcing, in contrast to the observational record. Overall, CM4.0 scores very well in its fidelity against observations compared to the Coupled Model Intercomparison Project Phase 5 generation in terms of both mean state and modes of variability and should prove a valuable new addition for analysis across a broad array of applications.
Balaji, V, E Maisonnave, Niki Zadeh, Bryan N Lawrence, Joachim Biercamp, Uwe Fladrich, G Aloisio, Rusty Benson, Arnaud Caubel, Jeffrey W Durachta, M-A Foujols, Grenville Lister, S Mocavero, Seth D Underwood, and Garrett Wright, January 2017: CPMIP: Measurements of Real Computational Performance of Earth System Models. Geoscientific Model Development, 10(1), DOI:10.5194/gmd-10-19-2017. Abstract
A climate model represents a multitude of processes on a variety of time and space scales; a canonical example of multi-physics multi-scale modeling. The underlying climate system is physically characterized by sensitive dependence on initial conditions, and natural stochastic variability, so very long integrations are needed to extract signals of climate change. Algorithms generally possess weak scaling and can be I/O and/or memory bound. Such weak-scaling, I/O and memory-bound multi-physics codes present particular challenges to computational performance.
Traditional metrics of computational efficiency such as performance counters and scaling curves do not tell us enough about real sustained performance from climate models on different machines. They also do not provide a satisfactory basis for comparative information across models.
We introduce a set of metrics that can be used for the study of computational performance of climate (and Earth System) models. These measures do not require specialized software or specific hardware counters, and should be accessible to anyone. They are independent of platform, and underlying parallel programming models. We show how these metrics can be used to measure actually attained performance of Earth system models on different machines, and identify the most fruitful areas of research and development for performance engineering.
We present results for these measures for a diverse suite of models from several modeling centres, and propose to use these measures as a basis for a CPMIP, a computational performance MIP.
Theurich, G, C DeLuca, T Campbell, F Liu, K Saint, M Vertenstein, J Chen, R C Oehmke, James D Doyle, T Whitcomb, A Wallcraft, M Iredell, T Black, A Da Silva, T Clune, R Ferraro, P Li, M Kelley, I Aleinov, V Balaji, and Niki Zadeh, et al., July 2016: The Earth System Prediction Suite: Toward a Coordinated U.S. Modeling Capability. Bulletin of the American Meteorological Society, 97(7), DOI:10.1175/BAMS-D-14-00164.1. Abstract
The Earth System Prediction Suite (ESPS) is a collection of flagship U.S. weather and climate models and model components that are being instrumented to conform to interoperability conventions, documented to follow metadata standards, and made available either under open source terms or to credentialed users.
The ESPS represents a culmination of efforts to create a common Earth system model architecture, and the advent of increasingly coordinated model development activities in the U.S. ESPS component interfaces are based on the Earth System Modeling Framework (ESMF), community-developed software for building and coupling models, and the National Unified Operational Prediction Capability (NUOPC) Layer, a set of ESMF-based component templates and interoperability conventions. This shared infrastructure simplifies the process of model coupling by guaranteeing that components conform to a set of technical and semantic behaviors. The ESPS encourages distributed, multi-agency development of coupled modeling systems, controlled experimentation and testing, and exploration of novel model configurations, such as those motivated by research involving managed and interactive ensembles. ESPS codes include the Navy Global Environmental Model (NavGEM), HYbrid Coordinate Ocean Model (HYCOM), and Coupled Ocean Atmosphere Mesoscale Prediction System (COAMPS®); the NOAA Environmental Modeling System (NEMS) and the Modular Ocean Model (MOM); the Community Earth System Model (CESM); and the NASA ModelE climate model and GEOS-5 atmospheric general circulation model.
We describe carbon system formulation and simulation characteristics of two new global coupled carbon-climate Earth System Models, ESM2M and ESM2G. These models demonstrate good climate fidelity as described in Part I while incorporating explicit and consistent carbon dynamics. The two models differ almost exclusively in the physical ocean component; ESM2M uses Modular Ocean Model version 4.1 with vertical pressure layers while ESM2G uses Generalized Ocean Layer Dynamics with a bulk mixed layer and interior isopycnal layers. On land, both ESMs include a revised land model to simulate competitive vegetation distributions and functioning, including carbon cycling among vegetation, soil and atmosphere. In the ocean, both models include new biogeochemical algorithms including phytoplankton functional group dynamics with flexible stoichiometry. Preindustrial simulations are spun up to give stable, realistic carbon cycle means and variability. Significant differences in simulation characteristics of these two models are described. Due to differences in oceanic ventilation rates (Part I) ESM2M has a stronger biological carbon pump but weaker northward implied atmospheric CO2 transport than ESM2G. The major advantages of ESM2G over ESM2M are: improved representation of surface chlorophyll in the Atlantic and Indian Oceans and thermocline nutrients and oxygen in the North Pacific. Improved tree mortality parameters in ESM2G produced more realistic carbon accumulation in vegetation pools. The major advantages of ESM2M over ESM2G are reduced nutrient and oxygen biases in the Southern and Tropical Oceans.
We describe the physical climate formulation and simulation characteristics of two new global coupled carbon-climate Earth System Models, ESM2M and ESM2G. These models demonstrate similar climate fidelity as the Geophysical Fluid Dynamics Laboratory’s previous CM2.1 climate model while incorporating explicit and consistent carbon dynamics. The two models differ exclusively in the physical ocean component; ESM2M uses Modular Ocean Model version 4.1 with vertical pressure layers while ESM2G uses Generalized Ocean Layer Dynamics with a bulk mixed layer and interior isopycnal layers. Differences in the ocean mean state include the thermocline depth being relatively deep in ESM2M and relatively shallow in ESM2G compared to observations. The crucial role of ocean dynamics on climate variability is highlighted in the El Niño-Southern Oscillation being overly strong in ESM2M and overly weak ESM2G relative to observations. Thus, while ESM2G might better represent climate changes relating to: total heat content variability given its lack of long term drift, gyre circulation and ventilation in the North Pacific, tropical Atlantic and Indian Oceans, and depth structure in the overturning and abyssal flows, ESM2M might better represent climate changes relating to: surface circulation given its superior surface temperature, salinity and height patterns, tropical Pacific circulation and variability, and Southern Ocean dynamics. Our overall assessment is that neither model is fundamentally superior to the other, and that both models achieve sufficient fidelity to allow meaningful climate and earth system modeling applications. This affords us the ability to assess the role of ocean configuration on earth system interactions in the context of two state-of-the-art coupled carbon-climate models.
This paper documents time mean simulation characteristics from the ocean and sea ice components in a new coupled climate model developed at NOAA's Geophysical Fluid Dynamics Laboratory (GFDL). The climate model, known as CM3, is formulated with effectively the same ocean and sea ice components as the earlier GFDL climate model, CM2.1, yet with extensive developments made to the atmosphere and land model components. Both CM2.1 and CM3 show stable mean climate indices, such as large scale circulation and sea surface temperatures (SSTs). There are notable improvements in the CM3 climate simulation relative to CM2.1, including a modified SST bias pattern and reduced biases in the Arctic sea ice cover. We anticipate SST differences between CM2.1 and CM3 in lower latitudes through analysis of the atmospheric fluxes at the ocean surface in corresponding Atmospheric Model Intercomparison Project (AMIP) simulations. In contrast, SST changes in the high latitudes are dominated by ocean and sea ice effects absent in AMIP simulations. The ocean interior simulation in CM3 is generally warmer than CM2.1, which adversely impacts the interior biases.
