The Implementation of Framework for Improvement by Vertical Enhancement Into Energy Exascale Earth System Model

Abstract

The low cloud bias in global climate models (GCMs) remains an unsolved problem. Coarse vertical resolution in GCMs has been suggested to be a significant cause of low cloud bias because planetary boundary layer parameterizations cannot resolve sharp temperature and moisture gradients often found at the top of subtropical stratocumulus layers. This work aims to ameliorate the low cloud problem by implementing a new computational method, the Framework for Improvement by Vertical Enhancement (FIVE), into the Energy Exascale Earth System Model (E3SM). Three physics schemes representing microphysics, radiation, and turbulence as well as vertical advection are interfaced to vertically enhanced physics (VEP), which allows for these processes to be computed on a higher vertical resolution grid compared to the rest of the E3SM model. We demonstrate the better representation of subtropical boundary layer clouds with FIVE while limiting additional computational cost from the increased number of levels. When the vertical resolution approaches the large eddy simulation‐like vertical resolution in VEP, the climatological low cloud amount shows a significant increase of more than 30% in the southeastern Pacific Ocean. Using FIVE to improve the representation of low‐level clouds does not come with any negative side effects associated with the simulation of mid‐ and high‐level cloud and precipitation, that can occur when running the full model at higher vertical resolution.

Introduction

Accurately representing clouds in weather and climate models is essential. Poor representation of clouds reduces our ability to determine the sign and magnitude of the cloud feedback in climate simulations and to predict temperature and precipitation in weather forecast models correctly. The low cloud bias, namely the lack of subtropical marine stratocumulus, is endemic to most global climate models (GCMs) and is mainly related to the strong low-level cloud sensitivity due to the cloud parameterization problem in climate models (Bony & Dufresne, 2005; Nam et al., 2012; Sherwood et al., 2014).

The Cloud Layers Unified By-Binormals (CLUBB) scheme is a higher order closure (HOC) that unifies the parameterization of the planetary boundary layer (PBL), shallow convection, and cloud macrophysics by using an assumed probability density function (PDF) to close on the clouds and turbulence (Golaz et al., 2007; Larson & Golaz, 2005; Larson et al., 2012). CLUBB predicts the mean state and second and third order turbulent moments of velocity and thermodynamic scalars, and it closes the system of equations by assuming a double Gaussian PDF composed with updraft and downdraft Gaussian PDFs. HOC models, including CLUBB, have recently been implemented into several GCMs (Bogenschutz et al., 2013; Cheng & Xu, 2015; Guo et al., 2014, 2015; Thayer-Calder et al., 2015) and to some degree have improved the representation of boundary layer clouds; for example, a more steady transition from the stratocumulus regime to the trade cumulus regime (Bogenschutz et al., 2013).

CLUBB has been known to perform best at high vertical resolution. Bogenschutz et al. (2012) showed that single column model (SCM) simulations with CLUBB improved the representation of the stratocumulus and transitional stratocumulus-over-cumulus regimes compared to traditional PBL and shallow convective parameterizations, and these improvements were most pronounced when high vertical resolution was used in the lower troposphere. Bogenschutz et al. (2021) (companion paper; henceforth B21) show that coarse vertical resolution in the Energy Exascale Earth System Model (E3SM; Golaz et al., 2019) is a significant cause of low cloud bias because CLUBB cannot resolve the sharp temperature and moisture gradients often found at the top of subtropical stratocumulus layers. B21 demonstrated that increasing the vertical resolution toward that which is typically used in large eddy simulation (LES) is a key ingredient to improving the representation of marine stratocumulus but comes with excessive computational cost. B21 also pointed out that the Zhang-McFarlane (ZM) deep convection scheme (G. J. Zhang & McFarlane, 1995) in E3SM is sensitive to higher vertical resolution and/or time step, resulting in degradations in the tropical climate and acts to partially negate the benefits of higher vertical resolution. Therefore, an intelligent method that uses higher vertical resolution to obtain optimal performance of a PBL scheme, while minimizing consequences to the climate skill metrics caused by other parameterizations in GCMs is desired. The method should negate both computational expense and circumvent running parameterizations which are not designed to run at such high vertical resolution.

We should note that the numerical choices implemented in CLUBB tend to smooth temporal and vertical variations in GCMs and could potentially contribute to the need for extremely high vertical resolution for improving marine stratocumulus clouds bias. Though we point out that each generation of GCMs tends to provide modest improvements in marine stratocumulus simulation with better parameterizations (Bogenschutz et al., 2013; Bretherton & Park, 2009), they all result in underrepresenting these stratocumulus clouds, even as these parameterizations become more sophisticated (Medeiros et al., 2012; Y. Zhang et al., 2019). The turbulence in subtropical marine stratocumulus clouds is generated at cloud top as a result of an abundance of radiative cooling occurring over a thin layer and sufficient turbulence is needed to sustain a subtropical stratocumulus boundary layer (Lilly, 1968). A coarse vertical grid in GCMs has the tendency to induce too much entrainment of dry free tropospheric air into the cloud layer and often results in reducing the cloud cover, which reduces the cloud top cooling feedback needed to sustain the cloud (Bretherton et al., 1999). Therefore, increasing vertical resolution in GCMs is an approach that should improve the marine stratocumulus cloud bias regardless of parameterization choice used in the model.

Yamaguchi et al. (2017) (henceforth Y17) have developed a method, the Framework for Improvement by Vertical Enhancement (FIVE), which focuses on running parameterizations, such as CLUBB, at higher vertical resolutions. The concept of FIVE is to create a separate computational domain, in which prognostic variables are allocated on a locally high-resolution grid. FIVE predicts prognostic variables by computing selected one-dimensional (1-D) processes on the locally high-resolution grid (e.g., microphysics, radiation, and turbulence) as well as applying interpolated tendencies from the host model for other processes. The host model predicts their prognostic variables by applying averaged tendencies computed on the locally high-resolution grid (Figure 1 in Y17). One advantage of FIVE is that high-resolution information is kept at all times during the simulation. Y17 also demonstrated that it is important to use FIVE to improve the representation of the large-scale vertical advection (i.e., to augment the vertical transport handled by the dynamical core of the host model). In Y17, the prototype FIVE demonstrated superior results for its application in SCM and two-dimensional regional model simulations compared to those performed with low vertical resolution in the host regional model. The prototype FIVE produced results comparable to those performed with a high vertical resolution regional model while saving computational cost.

In this study, we demonstrate that high vertical resolution for certain physical processes is a crucial component toward the improved climatological representation of low-level clouds in large-scale models such as E3SM. The purpose of this work is to implement FIVE into E3SM, which is also the first time that such a framework has been implemented into a global model. In addition to large-scale vertical advection, three physics schemes are interfaced with FIVE, which allows for these schemes to be computed on a higher vertical resolution grid compared to the rest of the E3SM model. A brief description of FIVE and E3SM, as well as numerical experiments, are given in Section 2. Simulated results are discussed in Section 3. A further discussion, including the importance of large-scale vertical advection in E3SM-FIVE, time step sensitivity, as well as future potential applications of FIVE, is given in Section 4. The summary is provided in Section 5.

Model Description and Numerical Experiments Framework for Improvement by Vertical Enhancement

FIVE predicts variables by computing selected 1-D processes (e.g., the parameterized cloud microphysics, radiation, and turbulence, and the resolved large-scale vertical advection) on the locally high-resolution grid and other processes on the host model grid. This is done by allocating prognostic variables in a separate memory (or computational domain) for the local high-resolution gird. In other words, the local high-resolution column is embedded in each host model column. For E3SM, horizontal wind components (U and V), temperature (T), and hydrometeors (Q) are prognostic variables. The embedded process calculations and predictions on the local high-resolution grid are called vertically enhanced physics (VEP). The VEP calculations do not interfere with the order of the computation of processes in the host model and the calculation processes are not repeated between the host model and VEP. For example, the T tendency profile in the host model is interpolated to the higher vertical resolution grid and then used to update T in VEP (i.e., $T_{\text{VEP}}^{n\ast }=T_{\text{VEP}}^n+\text{interpolation}\left(T_{\text{tendency}}\right)\mathrm{\Delta }t$, where $n$ is time step counter, $n\ast$ denotes a partial time step, and $\mathrm{\Delta }t$ is time step size). Then, $T_{\text{VEP}}^{n\ast }$ is further updated with selected physics schemes (e.g., turbulence, microphysics, and radiation scheme) on the high-resolution grid. The averages of the high-resolution tendencies computed in VEP are provided to the host model for prediction. In other words, the tendency in a layer of the host model matches a collection of thinner layers in VEP, and mass-weighted vertical averaging is used to aggregate the fine-resolution tendencies to the coarser grid (see Section 2.1 in Y17). The synchronization between the host model and VEP by exchanging tendencies with one another is necessary to prevent drift in the host model state. Because FIVE can keep any information in both host model and VEP states, they are conveniently used for tendency calculations. Figure 1 shows the schematic of the sequence of the processes between E3SM (the host model in this study) and selected physics schemes for VEP. More details of each process and how the variables are passed between the host model grid and the VEP finer grid in E3SM-FIVE are described in Sections 2.2 and 2.3.

Figure 1. Schematic of the sequence of the processes between the host model (i.e., Energy Exascale Earth System Model [E3SM]) and selected schemes for vertically enhanced physics (VEP). The sequence of processes in the host model is Zhang-McFarlane deep convection scheme (host model)→synchronization→Cloud Layers Unified By-Binormals (CLUBB) turbulence parameterization (VEP)→Morrison and Gettelman microphysics scheme version 2 (MG2) microphysics scheme (VEP)→(CLUBB and MG2 sub-cycle)→Rapid Radiative Transfer Model for GCMs (RRTMG) radiation schemes (VEP)→synchronization→Miscellaneous other atmospheric processes (host model)→horizontal advection (host model)→synchronization→vertical advection (VEP)→(horizontal advection and vertical sub-cycle)→final synchronization.

E3SM and the Selected Physics Schemes for VEP

The Department of Energy (DOE) E3SM coupled model version 1 has been released to the community and a detailed description is documented in Golaz et al. (2019). E3SM originated from a version of the Community Earth System Model version 1 (Hurrell et al., 2013) and the atmosphere component of E3SMv1, E3SM Atmosphere Model (EAM) (Rasch et al., 2019), is a descendant of the Community Atmosphere Model version 5.3 (CAM5.3) (Neale et al., 2010). EAM uses a spectral element (SE) dynamical core at a 110-km resolution on a cubed sphere geometry and a traditional hybridized sigma pressure vertical coordinate. The transition between terrain following and constant pressure coordinate is made at ∼200 hPa (∼11 km).

The vertical resolution in EAM is 72 layers with a top at approximately 60 km in altitude, which is higher than CAM5.3 with 30 vertical layers and a top at approximately 40 km in altitude. Fifteen layers reside between the surface and 850 hPa ($\mathrm{\Delta }Z\approx 25$ m at the surface and $\mathrm{\Delta }Z\approx 125$ m near 850 hPa) in EAM, with relatively finer vertical layers, compared to CAM5.3, with the goal to better capture thin clouds and sharp gradients at the top of the boundary layer. Between 850 and 500 hPa the vertical grid spacing is gradually increased from 100 to 500 m because strong water vapor gradients are frequently observed to occur at vertical scales of 500 m or less for important cloud features. This vertical resolution is needed for aerosol plume transport as well. Resolution from the free troposphere (above 500 hPa) up to the lower stratosphere (70 hPa) is increased from 600 to 1,200 m to allow for adequate representation of upward propagating large-scale tropical waves such as Kelvin and mixed-Rossby gravity.

Compared to CAM5.3, higher vertical resolution in EAM can better capture thin clouds, sharp gradients at the top of the boundary layer, rapid changes in process rates in microphysics and radiation (e.g., autoconversion, accretion, evaporation, and radiative heating rates), and cloud properties (e.g., drop size and rain rates); however, the underestimated liquid water content in marine stratocumulus still needs further improvement, which is a bias shared with most other modern GCMs. Despite the increases in vertical resolution in E3SM compared to CAM, B21 found that the vertical resolution on the order of 10 m is needed in the lower troposphere to resolve the sharp gradients often found at the top of the stratocumulus boundary layer. E3SM falls well below meeting this criterion. However, running at such high vertical resolution for all processes of E3SM is prohibitively expensive for long climate simulations and results in degradation of the climate simulation when running schemes not designed for high vertical resolution.

Y17 identified the essential processes which should be computed with high vertical resolution for successful stratocumulus simulations. In their study, they used a SCM framework to test microphysics, radiation, turbulence, and vertical advection (e.g., subsidence). Their results show that microphysics needs to be processed in VEP because it includes vertical transport in the form of cloud water sedimentation and rainwater precipitation. They also suggested computing vertical advection in VEP because the bias associated with subsidence (same as sedimentation) produces higher PBL depth, which results in a warmer and dryer PBL by entrainment. Turbulence parameterization in the host model resolution is too weak to mix the variability, so neglecting turbulence parameterization in VEP results in a particularly noisy profile in the host model. Using the turbulence parameterization in VEP can effectively smooth the variation developed in VEP. Radiation can be computed outside VEP provided that the interpolated radiative heating rate at the cloud top is accurately captured.

Following Y17, in addition to large-scale vertical advection discussed below, three physics schemes in EAM are selected for VEP to be run at higher vertical resolution to better represent low clouds:

1. Cloud Layers Unified By Binormals (CLUBB) is a third-order turbulence closure parameterization that unifies the treatment of PBL turbulence, shallow convection, and cloud macrophysics (Golaz et al., 2002; Larson & Golaz, 2005).

2. Morrison and Gettelman microphysics scheme version 2 (MG2) is a two-moment microphysics scheme to predict the number concentrations and mixing ratios of liquid and ice particles (Gettelman et al., 2015; Morrison & Gettelman, 2008).

3. Rapid Radiative Transfer Model for GCMs (RRTMG) longwave and shortwave radiation schemes use a modified correlated-k method to calculate radiative fluxes and heating rates in the clear sky and for condensed phase species (Iacono et al., 2008; Mlawer et al., 1997).

Before the start of these physics schemes in VEP, the tendency profile from E3SM (at the standard lower vertical resolution) is interpolated to the VEP vertical grid to obtain the synchronized tendency profile between E3SM and VEP for computing the process with the local high-resolution profiles (the first synchronization in Figure 1). In this study, the physics time step is 30 min for most parameterizations (i.e., deep convection) (Table 1). However, CLUBB and MG2 are subcycled together with a 5-min time step, as is done in EAMv1. The synchronization due to the ZM deep convection scheme is only called prior to the CLUBB and MG2 subcycle. Prognostic and diagnostic variables are calculated on the locally high-resolution grid and high-resolution information is kept at all times among the processes (i.e., turbulence, microphysics, and radiation). For example, cloud fraction is diagnosed by the CLUBB parameterization at high vertical resolution, which is saved in the EAM physics buffer (also known as “pbuf,” which is the data structure used to share information between parameterizations) and then passed to the microphysics and radiation parameterizations, instead of interpolating this variable back from the E3SM vertical grid. Finally, the averages of the high-resolution prognostic tendencies computed in VEP from the selected three physics schemes are provided to the host model for prediction. Note that the aerosol activation is calculated on the host model's coarser vertical grid in the current version of E3SM-FIVE. Cloud condensation nuclei calculated in the aerosol activation scheme is linearly interpolated to the VEP vertical grid before the microphysics process. In future versions of E3SM-FIVE, it would be possible to include aerosol activation in VEP to better represent aerosol-cloud interactions and assess the importance of this process at high vertical resolution.

Table 1 Principle Experiment Designs for This Study

FIVE runs	Layers	E3SM time step (seconds)	CLUBB and microphysics time step (seconds)
CNTL	72	1,800	300
FIVE_DOUB	92	1,800	300
FIVE_QUAD	132	1,800	300
FIVE_OCT	212	1,800	300
FIVE_16XL	372	1,800	300

Notes. The second column is the total vertical layers. The third and fourth column are the time step set up for E3SM dynamic and the time step for CLUBB and microphysics in each simulation run, respectively. All principle experiments are performed 5 years in length.

Abbreviations: CLUBB, Cloud Layers Unified By-Binormals; CNTL, control run; DOUB, double; E3SM, Energy Exascale Earth System Model; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

We note that we carefully checked that mass and energy are properly conserved when implementing FIVE into E3SM.

Large-Scale Vertical Advection

Besides the physics schemes, Y17 found it crucial that large-scale vertical advection be computed on the high-resolution grid. This is necessary to accurately balance entrainment via the turbulence scheme. Note that unlike the other processes, this calculation occurs in the dynamical core in EAM, which is computed after all physics processes (Figure 1). EAM uses hybrid sigma-pressure vertical coordinate and the vertically Lagrangian approach from Lin (2004). All prognostic variables are defined on the mid-points of the grid layer. These prognostic variables (U, V, T, and Q) advanced in time on a moving vertical coordinate system as a floating point. At the end of the simulation cycle, these variables are remapped back to the mid-points of the grid layer using the monotonic remapping algorithm based on Zerroukat et al. (2005). E3SM-FIVE proceeds as follows: after the horizontal advection, miscellaneous processes and horizontal advection tendency for U, V, T, and Q are interpolated from the host model to the VEP vertical grid and the prognostic variables in VEP are updated with the net tendency, like we did in the first synchronization (Figure 1). Once prognostic variables are vertically transported with the Lagrangian scheme in VEP, the prognostic variables are remapped back to the VEP mid-points of the grid layer for large-scale vertical advection adjustment (i.e., remapping process) with the existing remapping algorithm in E3SM. In the final synchronization, the mass-weighted vertical averaging is used to aggregate the fine-resolution remapped U, V, T, and Q to the coarser host grid. Note that there are two 15-min remapping time step per 30-min time step in E3SM-FIVE, so the processes in the dynamical core (horizontal advection (E3SM)→synchronization from E3SM to VEP→remapping (VEP)→synchronization from VEP to E3SM) go through twice. The importance of large-scale vertical advection in FIVE for E3SM is discussed in Section 4.1.

Model Configuration and Numerical Experiment Design

The purpose of this experiment design is to determine whether the representation of marine stratocumulus is improved when the vertical resolution in VEP increases for the selected physical processes (i.e., CLUBB, MG microphysics scheme, and RRTMG radiation scheme) and large-scale vertical advection. Note that all other processes are computed on the standard 72-layer grid. The control model (CNTL) is based on the configuration of E3SMv1, 110-km horizontal resolution (ne30), and 72 vertical layers. Four principle simulations are designed to double (FIVE_DOUB), quadruple (FIVE_QUAD), octuple (FIVE_OCT), and sexdecuple (FIVE_16XL) the vertical resolution of VEP between 995 and 700 hPa (Table 1). The vertical grid configurations for VEP are identical to the grid configuration of the E3SM benchmark experiments (DOUB, QUAD, OCT, and 16XL) in B21 (companion paper), where vertical resolution is increased in the lower troposphere for the entire model. The vertical grid spacings in the lower troposphere for OCT and 16XL are similar to those typically used for LES studies (Stevens et al., 2005; van der Dussen et al., 2013). The comparison of E3SM benchmark experiments and E3SM-FIVE runs is presented in Section 3.3. Similar to B21, none of our FIVE experiments are tuned in any manner.

Although a time step reduction is necessary for a stable benchmark OCT run in B21, FIVE_OCT does not need a time step reduction. To help elucidate any sensitivities arising from time step differences between FIVE simulations and benchmark runs, two additional simulations, FIVE_OCT_t150 and FIVE_OCT_d900, are performed. We reduced the CLUBB and microphysics time step from 300 to 150 s for FIVE_OCT_t150 and the dynamics time step from 1,800 to 900 s for FIVE_OCT_d900, which is the same time step set up as the benchmark OCT experiment. Note that the dynamics time step remains unmodified, relative to CNTL, for all simulations besides FIVE_OCT_d900 (Table 2).

Table 2 Sensitivity Experiment Designs for This Study

FIVE runs	Layers	E3SM time step (seconds)	CLUBB and microphysics time step (seconds)
FIVE_OCT_t150	212	1,800	150
FIVE_OCT_d900	212	9,00	150
FIVE_OCT_noLS	212	1,800	300

Notes. The second column is the total vertical layers. The third and fourth column are the time step set up for E3SM dynamic and the time step for CLUBB and microphysics in each simulation run, respectively. All sensitivity experiments are performed 2 years in length.

Abbreviations: CLUBB, Cloud Layers Unified By-Binormals; E3SM, Energy Exascale Earth System Model; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple.

Another simulation, FIVE_OCT_noLS, is designed as a sensitivity test for the effects of large-scale vertical advection on the high vertical resolution grid. FIVE_OCT_noLS means no large-scale vertical advection is computed in FIVE (i.e., it is computed on the standard 72-layer grid), but the three selected physics schemes remain coupled to FIVE. The duration of all principle experiments is 5 years, while the sensitivity runs (FIVE_OCT_t150, FIVE_OCT_d900, and FIVE_OCT_noLS) are integrated for 2 years. Tables 1 and 2 summarize the grid configuration and time step settings for our principle and sensitivity experiments.

Results E3SM Control Run

As previously mentioned, E3SMv1 has higher vertical resolution compared to CAM5.3, with the expectation that it would better represent marine stratocumulus. However, the stratocumulus biases are similar in the two models, so further improvements to the low-level cloud amount and shortwave cloud radiative effect (SWCRE) biases are needed in E3SM. Figure 2a shows the climatologically averaged low-level cloud amount from Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) lidar data from January 2007 to January 2010. Low stratiform clouds are primarily found over the oceans and those clouds can be classified into three types of stratiform clouds by Klein and Hartmann (1993): stratiform clouds on the east side of the oceanic subtropical highs, stratocumulus clouds over the warm western boundary currents in winter, and Arctic stratus. For this study, we are primarily concerned with the stratiform clouds on the east side of the oceanic subtropical highs.

Figure 2. (a) Low level cloud amount from Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) lidar data from January 2007 to January 2010. (b) Averaged low cloud amount in the control run (CNTL). (c) The differences of low level cloud amount between CNTL and the observation.

To ensure an apples-to-apples comparison when evaluating simulated low cloud climatology with satellite observations, our E3SM simulations use the Cloud Feedback Model Intercomparing Project Observation Simulator Package (Bodas-Salcedo et al., 2011). Compared to CALIPSO, CNTL captures the general pattern of low-level cloud amount (Figure 2b), and the correlation between CNTL and the observation can be as high as 0.87. The underestimated low-level cloud amount in CNTL mainly appears in the tropical and subtropical regions, where the biases over eastern oceans, for example, Eastern Pacific Ocean, Eastern Atlantic Ocean, and Eastern Indian Ocean, can be more than a 30% deficit (Figure 2c). The stratiform clouds over these regions occur in response to trade winds blowing from mid-latitudes toward the intertropical convergence zone (ITCZ). These clouds form over oceans with relatively cool sea surface temperature, associated with ocean upwelling circulation, and form a strong temperature inversion that caps the boundary layer. As the air in the trade winds approaches the ITCZ and warmer water, the trade inversion generally rises and weakens, and trade wind cumulus convection replaces the stratiform clouds.

Due to the underestimated low-level cloud amount in CNTL, the SWCRE biases also appear over the corresponding areas of eastern oceans compared to the observation (Figure 3c). The observational data of SWCRE is from the Clouds and the Earth's Radiant Energy System Energy Balanced and Filled top-of-atmosphere (TOA) data product averaged from 2000 to 2015 (Figure 3a). It should be noted that the maximum SWCRE biases are governed by not only low cloud amount but also solar insolation, so that the variation of SWCRE is quite high from season to season.

Figure 3. (a) Shortwave cloud radiative effect (SWCRE) from the Clouds and the Earth's Radiant Energy System (CERES) Energy Balanced and Filled (EBAF) top-of-atmosphere (TOA) data product averaged from 2000 to 2015. (b) Averaged SWCRE in the control run (CNTL). (c) The differences of SWCRE between CNTL and the observation.

E3SM-FIVE Results

In this section, we primarily focus on assessing the behavior of low clouds in our various E3SM-FIVE configurations and how they compare to the control run (CNTL). Figure 4 shows that compared to CNTL, the biases associated with low-level cloud amount are gradually improved as we increase the VEP vertical resolution in our E3SM-FIVE simulations. The improvement is most pronounced in the eastern Pacific Ocean. It is interesting to note that the improvement of the low-level cloud amount in FIVE_DOUB is negligible, while modest improvements are seen in FIVE_QUAD with increases of the low-level cloud amount around 5%–10% in the tropical and subtropical regions. When the vertical resolution approaches LES-like resolutions in the FIVE_OCT and FIVE_16XL experiments, the low-level cloud amount is significantly increased.

Figure 4. (a) The differences of low-level cloud amount between FIVE_DOUB and CNTL (Figure 2b). (b–d) are the same as (a) but for FIVE_QUAD, FIVE_OCT, and FIVE_16XL, respectively. CNTL, control run; DOUB, double; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

It is important to note that the reduction of low cloud amount biases with increasing vertical resolution is consistent with the results of the companion study (Figure 3 in B21), which found that LES-like vertical resolution is necessary to achieve significant bias reductions in the low cloud climatology. Compared to CNTL, the low-level cloud amounts in FIVE_OCT and FIVE_16XL are increased by more than 30% in the southeastern Pacific Ocean. The improvement of low cloud biases for offshore stratocumulus (or “core” regions as defined in Klein & Hartmann, 1993) appears to mostly converge with vertical resolution between FIVE_OCT and FIVE_16XL simulations. B21 was not able to address whether their 16XL simulation led to better results compared to their OCT simulation because their 16XL simulation required extreme time step adjustment, which introduced a large sensitivity that made it impossible to determine effects from vertical resolution alone (discussed in Section 3.3). Since E3SM-FIVE does not require time step reduction, we can determine that going from LES-like vertical resolutions of FIVE_OCT to FIVE_16XL does not appear to lead to significant improvements for offshore stratocumulus but appears to lead to some improvements for coastal low-level cloud amount (Figure 4d).

The SWCRE biases are also gradually improved in the corresponding marine stratocumulus areas as the VEP resolution increases, which is most pronounced in the southeast Pacific Ocean (Figure 5). Our simulations show that the improved SWCRE biases associated with low clouds is more prominent in offshore “core” regions compared to the coasts. However, the maximum SWCRE biases associated with low-level cloud in CNTL occurs in the coastal areas, such as the west coast of North America and South America (Figure 2c). Only FIVE_16XL demonstrates a visible improvement of SWCRE biases along the coasts (Figure 5d). Therefore, our result shows that increasing vertical resolution toward LES-like vertical resolutions has potential to improve the simulation of stratocumulus along the coastal regions, which are typically areas of stubborn biases in GCMs. It is likely that concurrent increases in horizontal resolution are required to substantially reduce these coastal biases.

Figure 5. (a) The differences of shortwave cloud radiative effect between FIVE_DOUB and CNTL (Figure 3b). (b–d) are the same as (a) but for FIVE_QUAD, FIVE_OCT, and FIVE_16XL, respectively. CNTL, control run; DOUB, double; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

Beyond the subtropical regions, it is interesting to point out that FIVE_16XL predicts less low-level cloud amount over the polar regions than CNTL and the other FIVE configurations. The SWCRE over the polar regions in FIVE_16XL is also correspondingly affected compared to the other simulations (Figure 5d). B21 (companion study) presented a similar feature in the benchmark OCT run and speculated a potential sensitivity of CLUBB, MG2, or their interactions to high vertical resolution in the presence of mixed phase clouds and/or the stable boundary layer (Figures 3 and 6 in B21). In their benchmark simulations, OCT resulted in significant differences in liquid water path (LWP) and ice water path (IWP) in the polar regions compared to other lower vertical resolution benchmark cases. Higher LWP and lower IWP in the Antarctic Circle (∼60°S in latitude) and lower LWP but relatively steady IWP in the north polar regions in OCT (Figure 12 in B21) contribute to a slightly weaker SWCRE in the polar regions, which is closer to the observation. In our simulations, both LWP and IWP in FIVE_16XL are lower than those in other E3SM-FIVE simulations (Figures 6a and 6b), which weakens not only SWCRE but also longwave cloud radiative effect (LWCRE) in the polar regions (Figures 6c and 6d). Among the E3SM-FIVE and E3SM benchmark simulations, the globally averaged SWCRE and LWCRE for FIVE_16XL is most comparable to the observation (Table 6). However, the weaker SWCRE in FIVE_16XL compensates the negative biases in the polar regions of CNTL against the observation (Figure 3c). Figure 7 shows the differences of SWCRE between E3SM-FIVE simulations and the observation. Overall, the results in FIVE_16XL show the most improvement compared to other FIVE principle simulations (Figure 7d). The sensitivity to vertical resolution in the polar regions is interesting but beyond the scope of this work and would be interesting to pursue in future studies.

Figure 6. The zonal average of (a) cloud liquid water path (kg/m2), (b) cloud ice water path (kg/m2), (c) shortwave cloud radiative effect (W/m2), and (d) longwave cloud radiative effect (W/m2) from the simulations of Energy Exascale Earth System Model (E3SM)-Framework for Improvement by Vertical Enhancement (FIVE). CERES-EBAF is the Clouds and the Earth's Radiant Energy System (CERES) Energy Balanced and Filled (EBAF) top-of-atmosphere (TOA) data product averaged from 2000 to 2015. CNTL, control run; DOUB, double; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

Figure 7. (a) The differences of shortwave cloud radiative effect between FIVE_DOUB and the observation (Figure 3a). (b–d) are the same as (a) but for FIVE_QUAD, FIVE_OCT, and FIVE_16XL, respectively. DOUB, double; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

The primary focus of this study is the improvements on the subtropical low clouds due to increased vertical resolution for select processes. Besides presenting the effects on the global low cloud climatology, we also focus on the five subtropical marine stratus regions for detailed analyses. Based on the definition of stratus regions in Klein and Hartmann (1993), Table 3 shows the selected regions, their locations, and the seasons of maximum stratus that we analyze.

Table 3 The Five Status Regions, the Season of Maximum Stratiform Clouds, and Their Geographical Location (Core Area) Referred to the Definition in Klein and Hartmann (1993)

Region	Season of maximum stratus	Location (core area)	Location (extended area)
Peruvian	SON	10°–20°S, 90°–100°Wa	5°–35°S, 80°–110°W
Californian	JJA	20°–30°N, 120°–130°W	10°–40°N, 116°–145°W
Namibian	SON	10°–20°S, 0°–10°E	5°–35°S, 15° W–15°E
Australian	DJF	25°–35°S, 95°–105°E	/
Canarian	JJA	15°–25°N, 25°–35°W	/

Notes. SON indicates September, October, and November, and so on. The extended area is defined for the analysis in Table 6.

^aLocation of the Peruvian region was defined to 0°–20°S, 80°–90°W in Klein and Hartmann (1993).

Figures 8a and 8b show that the cloud fraction and cloud liquid amount in the Peruvian region increase along with the total number of vertical layers in the E3SM-FIVE simulations, while the climatological cloud top height and cloud thickness both increase as well. The maximum cloud fraction in CNTL resides at ∼880 hPa, while the peak of the cloud fraction profile in all E3SM-FIVE simulations is about 20 hPa higher (∼860 hPa). Compared to the observations, all E3SM-FIVE experiments simulate too little cloud fraction and too thin cloud depth; however, they produce a peak cloud liquid water amount that is fairly comparable to the observations. Here, the observational data is provided by CALIPSO, CloudSat, and Moderate Resolution Imaging Spectroradiometer in a merged product called C3M (Kato et al., 2010). The minimum peak of the longwave cloud heating rate profile in all E3SM-FIVE simulations is also 20 hPa higher than in CNTL (Figure 8c), which is similar to the profiles of the benchmark runs presented in B21.

Figure 8. Spatial- and temporal-averaged profiles of (a) cloud fraction, (b) cloud liquid water amount, (c) longwave heating rate, (d) second moment vertical velocity (w′2¯), and (e) third moment vertical velocity (w′3¯) in the Peruvian region (defined in Table 3) from the simulations of Energy Exascale Earth System Model (E3SM)-Framework for Improvement by Vertical Enhancement (FIVE). (f–j) are the same as (a–e) but in the Californian region. (k–o) are the same as (a–e) but in the Namibian region. (p–t) are the same as (a–e) but in the Australian region. (u–y) are the same as (a–e) but in the Canarian region. CNTL, control run; DOUB, double; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

It is worthwhile to mention that FIVE_DOUB and FIVE_QUAD have similar results compared to the profiles of the benchmark DOUB and QUAD in B21 over the Peruvian region. However, the peak cloud fraction and cloud liquid amount are predicted ∼30% higher in OCT than FIVE_OCT. The third moment of vertical velocity ($\overline{w{\prime }^3}$) demonstrates the model's ability to realistically simulate the transition from stratocumulus to cumulus. In CNTL, the $\overline{w{\prime }^3}$ profile has the highest positive magnitude, which is indicative of a positive vertical velocity skewness (Figure 8e). Usually, where trade wind cumulus is present, $\overline{w{\prime }^3}$ increases in positive magnitude and develops a bimodal vertical structure representative of a decoupled boundary layer. The profile of the third moment of vertical velocity in all E3SM-FIVE simulations is similar to the profile in CNTL but smaller in magnitude. Overall, profiles in the Peruvian region show that E3SM-FIVE with higher vertical resolution can simulate a thicker cloud deck that is more turbulent and somewhat less decoupled.

Our global analysis shows that the stratocumulus in the Californian region is more resistant to change with increases in vertical resolution, and that conclusion is reflected when analyzing the profiles. The cloud fraction over the Californian region decreases in FIVE_DOUB and then increases along with the vertical resolution in the E3SM-FIVE simulations, while the cloud top height and cloud thickness both increase as well (Figure 8f). The peak of cloud liquid amount in all E3SM-FIVE simulations tends to be 20–30 hPa higher than the peak in the observations (Figure 8g). Compared to CNTL, FIVE_16XL is the only simulation showing some improvement in the cloud fraction and cloud liquid amount and this is associated with a stronger longwave cloud cooling rate (Figure 8h). We point out that the peak magnitude of cloud fraction and cloud liquid water as well as the cloud top height in FIVE_16XL is similar to the result in the benchmark OCT for this region (Figure 9 in B21), potentially suggesting that E3SM-FIVE requires more vertical levels to obtain improvements relative to the benchmark simulations. In Section 4, we discuss possible reasons why E3SM-FIVE cannot fully replicate the results of the high-resolution benchmarks presented in B21. Besides FIVE_OCT, the $\overline{w{\prime }^3}$ profiles over the Californian region in other E3SM-FIVE simulations do not show a clear bimodal vertical structure, including CNTL (Figure 8j). However, FIVE_DOUB, FIVE_QUAD, and FIVE_16XL have a higher positive magnitude in $\overline{w{\prime }^3}$ compared to CNTL, likely helping to hinder increases in cloud as resolution increases.

The improvements in cloud fraction over the Namibian region (Figure 8k), which is a fairly active and strong stratocumulus regime, are similar to the Peruvian region (cf. Figure 8a). In CNTL, the $\overline{w{\prime }^3}$ profile over the Namibian region has the highest positive magnitude (Figure 8o), similar to the profile in the Peruvian region (Figure 8e). However, while all E3SM-FIVE runs exhibit a lower magnitude of $\overline{w{\prime }^3}$ than the CNTL run, there is an evidence of the boundary layer becoming more decoupled as vertical resolution increases. The results for the Australian region and the Canarian region also show improvements when FIVE is used compared to CNTL; though perhaps relatively more muted. It is likely because these regions are not characterized by strong inversions relative to the others and suffer from less severe low-cloud biases, hence subjected to somewhat less sensitivity to vertical resolution.

The Comparison of E3SM-FIVE and E3SM Benchmarks Computational Cost

B21 gradually increased the vertical resolution for the entire E3SM model from 135 to 15 m near the climatologically typical stratocumulus inversion height, using the same vertical grid designs as this study. Based on results from LES studies, 5–10 m vertical resolution is recommended to resolve the inversion (Bretherton et al., 1999; Stevens et al., 2005). The benchmark simulations in B21 show that the improvement of low cloud biases has become conspicuous only when the vertical resolution approaches the resolutions representative of LES. The improvement of low cloud biases in DOUB (70 m vertical resolution) was negligible, while marginal impacts were seen in QUAD (35 m vertical resolution) for low cloud biases, especially in the southeastern Pacific Ocean and the southeastern Atlantic Ocean.

Increasing vertical resolution is a necessary ingredient to improve the low cloud amount; however, using LES-like vertical resolution for the entire model is expensive. Table 4 shows the comparison of computational cost between the E3SM benchmarks and the E3SM-FIVE simulations. The computational cost of the benchmark runs is exponentially increased with the total number of layers, partially due to the fact that the OCT and 16XL benchmark runs require a reduction of time step.

Table 4 The Comparison of Computational Cost Between Simulations in E3SM-Benchmarks and E3SM-FIVE

Benchmarks	CNTL	DOUB	QUAD	OCT	16XL
E3SM-Benchmarks (SYPD/1,024 cores)	4.3	2.2	1.2	0.23	0.03
E3SM-FIVE (SYPD/1,024 cores)	N/A	1.8	1.6	1.21	0.67

Note. SYPD indicates simulated years per day.

Abbreviations: CNTL, control run; DOUB, double; E3SM, Energy Exascale Earth System Model; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

In comparison, running FIVE_DOUB is slightly slower than running DOUB. It should be noted that the current prototype version of E3SM-FIVE has not yet been optimized and doing so in the future would likely yield a reduction to the overhead costs of E3SM-FIVE. However, FIVE_QUAD is run with the same time step settings as QUAD and runs slightly faster. In FIVE_QUAD, the overhead cost of FIVE is not as large as the expense of running horizontal advection and other high vertical resolution physics schemes, which are not computed in FIVE (e.g., deep convection scheme). Furthermore, a significant performance advantage is found when running at LES-Like vertical resolutions in E3SM-FIVE, which is partially because no time step reduction is required in any E3SM-FIVE simulations (Table 1). FIVE_OCT is about four times faster than OCT, while the savings of FIVE_16XL is more than an order of magnitude than 16XL.

These timing numbers represent a significant advantage for E3SM-FIVE. For instance, B21 was unable to run their 16XL experiment for longer than 2 years, while we are able to report on a 5-year simulation of FIVE-16XL without undue computational burden.

Comparison of Climatology

Figure 9 shows the differences of low-level cloud amount between the E3SM benchmarks and the E3SM-FIVE experiments. Compared to the E3SM-FIVE simulations, the increases of low-level cloud amount in the benchmark simulations are more significant as the vertical resolution increases. However, both the benchmark and E3SM-FIVE simulations show improvement in the simulated low-cloud amount as the vertical resolution increases by comparing these simulations with the observations. We want to highlight that the benchmark OCT run overestimated the low-level cloud amount in the offshore region of Peru by 20%–25% (Figure 2 in B21), which results in overly reflective clouds.

Figure 9. (a) The differences of low-level cloud amount between Energy Exascale Earth System Model (E3SM) benchmark DOUB and FIVE_DOUB (Figure 1b). (b) is the same as (a) but between QUAD and FIVE_QUAD. (c) is the same as (a) but between OCT and FIVE_OCT. DOUB, double; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

Table 5 shows the root mean square error (RMSE) and the bias of low-level cloud amount for three extended stratocumulus regions defined in Table 3 in each benchmark and E3SM-FIVE run against the observations. In terms of RMSE, the three regions generally show increasing skill in the benchmark simulations for each region as vertical resolution increases, while the OCT simulation performs best for all regions. Besides FIVE_16XL, E3SM-FIVE simulations follow the trend of increasing skill as the vertical resolution increases. For the Peruvian and Namibian regions, the OCT simulation is an outlier as it shows a net positive bias, which is not seen in any E3SM-FIVE simulations. Compared to the benchmark simulations, E3SM-FIVE simulations generally have higher RMSE and bias scores for low cloud amount in the stratocumulus regions, though the low cloud climatology is still improved for these regions when compared to E3SM CNTL.

Table 5 RMSE and Bias Computed Relative to the CALIPSO Observations for the Low Cloud Amounts for Three Extended Stratocumulus Regions (Table 3) for Each Experiment in E3SM Benchmarks and E3SM-FIVE Against the Observations

	E3SM-benchmarks				E3SM-FIVE
	CNTL	DOUB	QUAD	OCT	DOUB	QUAD	OCT	16XL
RMSE
Peruvian	23.8	19.2	19.3	10.2	21.0	18.2	14.0	16.9
Californian	26.2	24.8	19.2	12.3	27.6	22.6	19.6	16.4
Namibian	20.5	18.9	7.5	7.3	20.3	14.0	11.4	12.1
Bias
Peruvian	−19.2	−14.5	−12.2	2.6	−18.7	−13.7	−8.4	−12.4
Californian	−22.5	−19.8	−14.8	−5.7	−24.8	−19.5	−15.9	−14.0
Namibian	−19.0	−14.9	−2.5	2.7	−18.0	−10.8	−6.5	−8.3

Abbreviations: CALIPSO, Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation; CNTL, control run; DOUB, double; E3SM, Energy Exascale Earth System Model; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; RMSE, root mean squared errors; 16XL, sexdecuple.

The global RMSE of low-level cloud amount for each benchmark and E3SM-FIVE simulation is listed in Table 6. When the vertical resolution increases, both the benchmark and E3SM-FIVE simulations show a declining trend of RMSE for low-level cloud amount. Overall, however, the benchmark simulations have smaller errors when compared to the E3SM-FIVE simulations. While the E3SM-FIVE runs overall exhibit quantitatively similar behavior in regard to the representation of low cloud climatology when compared to benchmark runs (i.e., as vertical resolution increases, low cloud amount increases), it is interesting to question why differences exist between these two sets of experiments. We hypothesize that two factors are likely contributing toward these differences. First, differences in the simulated Hadley circulation due to feedbacks from not running the ZM deep convection scheme at high vertical resolution in E3SM-FIVE is likely acting to effect low cloud simulation in the descending branch. Second, and likely the more significant reason, the errors that are associated with the tendency interpolation for synchronization between E3SM and VEP that causes a loss accuracy versus in the free running simulation.

Table 6 RMSE Biases of Low-Level Cloud Amount (%), Mid-Level Cloud Amount (%), High-Level Cloud Amount (%), SWCRE (W/m²), LWCRE (W/m²), and Precipitation (mm/day) for Each Experiment in E3SM Benchmarks and E3SM-FIVE Against the Observations

	E3SM-benchmarks				E3SM-FIVE
	CNTL	DOUB	QUAD	OCT	DOUB	QUAD	OCT	16XL
Low-level cloud amount (%)	12.75	11.90	11.21	10.18	12.73	11.50	11.36	11.01
Mid-level cloud amount (%)	7.32	7.31	7.52	7.96	7.28	7.26	7.28	7.13
High-level cloud amount (%)	7.87	8.00	7.99	9.00	7.95	7.91	7.84	7.81
SWCRE (W/m2)	9.54	9.50	8.98	9.31	9.72	9.45	9.35	8.63
LWCRE (W/m2)	8.43	8.19	8.04	9.00	8.84	8.24	8.05	8.06
Precipitation (mm/day)	1.06	1.14	1.36	1.66	1.12	1.15	1.14	1.15

Abbreviations: CALIPSO, Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation; CNTL, control run; DOUB, double; E3SM, Energy Exascale Earth System Model; FIVE, Framework for Improvement by Vertical Enhancement; LWCRE, longwave cloud radiative effect; OCT, octuple; QUAD, quadruple; RMSE, root mean squared errors; SWCRE, shortwave cloud radiative effect; 16XL, sexdecuple.

Compared to the observation, the RMSE of SWCRE in the benchmark experiments also show a trend toward improvement when the vertical resolution increases (Table 6). However, when the vertical resolution approaches 15 m in OCT, the errors of SWCRE increase again. This rebound trend of RMSE for SWCRE does not appear in E3SM-FIVE. Although the RMSE of SWCRE in the E3SM-FIVE runs are generally higher than that in the benchmark simulations, the fact that errors get smaller as vertical resolution increases is encouraging. Figure 10 shows the differences of SWCRE between the E3SM benchmarks and the E3SM-FIVE experiments. In general, FIVE_OCT has more reflective clouds compared to OCT (Figure 10c). As mentioned previously, the overestimated low-level cloud amount and SWCRE over the offshore region of Peru are both present in OCT. Overall, FIVE_OCT compares better with the observations in this region (Figures 9c and 10c). B21 found that SWCRE is too weak in the OCT simulation over the tropical regions and also reported that the ZM deep convection scheme is sensitive to the higher vertical resolution and/or time step, which results in a degraded climate simulation over the deep convective tropics. Since E3SM-FIVE does not run the ZM deep convection scheme at high vertical resolution, we can avoid the negative consequences of running parameterizations that may not be designed to run at such high vertical resolution, which is another major benefit of FIVE (Figure 7).

Figure 10. (a) The differences of shortwave cloud radiative effect between Energy Exascale Earth System Model (E3SM) benchmark DOUB and CNTL (Figure 3b). (b) is the same as (a) but between QUAD and FIVE_QUAD. (c) is the same as (a) but between OCT and FIVE_OCT. CNTL, control run; DOUB, double; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

Compared to observations, the RMSE of precipitation for the E3SM-FIVE runs remains steady as the vertical resolution increases. This is very different when compared to the benchmark simulations, where the precipitation RMSE increases significantly as the vertical resolution increases, likely due to the sensitivity of the deep convection scheme to high vertical resolution (Table 6). We further demonstrate this by comparing the geographical biases for the FIVE_OCT and OCT simulations. Figure 11 shows that compared to the precipitation biases in FIVE_OCT, the biases of precipitation in OCT are higher in the tropical regions. B21 found that when the vertical resolution increases, the large-scale precipitation rate gradually increases and the convective precipitation rate declines (Figure 11 in B21). With their analysis, it was not clear if this shift in partitioning of precipitation type and degradation of precipitation skill scores with increasing vertical resolution represented a sensitivity coming from the ZM deep convection scheme itself or due to a sensitivity arising from the CLUBB and/or microphysical parameterization. In the E3SM-FIVE simulations, large-scale precipitation rate slightly increases when the vertical resolution increases, but no obvious sensitivity is found in regard to the convective precipitation rate (Figure 12). This suggests that the strong sensitivity and poor skill scores demonstrated by the OCT simulation in B21 stems from a sensitivity of the ZM deep convection scheme due to vertical resolution and/or time step rather than a sensitivity arising in the CLUBB turbulence scheme and/or the MG2 microphysics scheme.

Figure 11. (a) Precipitation from Global Precipitation Climatology Project (GPCP) data set averaged from 1979 to 2014. (b) The differences of precipitation between Energy Exascale Earth System Model (E3SM) benchmark octuple (OCT) and the observation. (c) The differences of precipitation between FIVE_OCT and the observation. The bottom two rows of (b and c) display the evolution of the geographical biases, root mean squared errors (RMSE), and correlation coefficients (CORR) of the OCT and FIVE_OCT simulations computed relative to the observation.

Figure 12. The zonal average of (a) convective precipitation rate (mm/day) and (b) large-scale precipitation rate (mm/day) from the simulations of Energy Exascale Earth System Model (E3SM)-Framework for Improvement by Vertical Enhancement (FIVE). CNTL, control run; DOUB, double; OCT, octuple; QUAD, quadruple; 16XL, sexdecuple.

The RMSE of mid-level cloud amount, high-level cloud amount, and LWCRE for each E3SM benchmark and E3SM-FIVE experiment against the observations are also presented in Table 6. Although the benchmark runs improve low-level cloud amount compared to CNTL, the RMSE of mid- and high-level cloud amount degrade with higher vertical resolution. On the other hand, while E3SM-FIVE improves low-level cloud amount compared to CNTL, and in a similar manner compared to benchmarks, the skill scores of mid- and high-level cloud amount are not negatively impacted. Similar to the RMSE of SWCRE, the RMSE of LWCRE in the benchmark experiments shows improvement when the vertical resolution increases toward QUAD, but the errors of LWCRE increase again in OCT. This does not appear in our E3SM-FIVE simulations as the RMSE remains steady at the higher vertical resolution configurations. Our results show global skill of both LWCRE and SWCRE do not exhibit degradation with respect to vertical resolution in E3SM-FIVE because these simulations are not subjected to sensitivities of the ZM scheme at high vertical resolution.

We did not discuss climate sensitivity in this study, but Golaz et al. (2019) shows that E3SM has high climate sensitivity. Based on Cess et al. (1990), the ratio of net cloud radiative effect to net radiative flux can roughly represent the climate model sensitivity in responding cloud radiative effect. In our results, the biases of net radiative flux at TOA against the observations decrease from 1.91 W m⁻² in CNTL to −0.34 W m⁻² in FIVE_OCT while the vertical resolution increases. The biases of net cloud radiative effect get larger (stronger cooling effect) from −4.15 W m⁻² in CNTL to −5.40 W m⁻² in FIVE_OCT (Table 7). The estimated ratios show that the net radiative flux at TOA changes significantly in response to the changes of net cloud radiative effect with higher vertical resolution. It also means that the climate sensitivity may increase in E3SM after FIVE is introduced. The radiation budgets at TOA are also shown in Table 7. The result of FIVE_16XL is different from other simulations because of cloud radiative effect in the polar regions (Figure 7), which needs further investigation. Future work should also more astutely address climate sensitivity in an E3SM configuration that can better resolve marine stratocumulus.

Table 7 Net Shortwave Flux (W/m²), Upwelling Longwave Flux (W/m²), Net Radiative Flux (W/m²), and Net Cloud Radiative Effect (W/m²) at TOA for Experiments Performed

	CNTL	FIVE_DOUB	FIVE_QUAD	FIVE_OCT	FIVE_16XL
Net shortwave flux at TOA (W/m2)	244.13 (3.02)	243.78 (2.66)	242.08 (0.96)	241.50 (0.38)	244.15 (3.03)
Upwelling longwave flux at TOA (W/m2)	241.49 (1.24)	241.43 (1.17)	241.30 (1.04)	241.11 (0.86)	241.39 (1.14)
Net radiative flux at TOA (W/m2)	2.77 (1.91)	2.48 (1.62)	0.91 (0.05)	0.52 (−0.34)	2.91 (2.05)
Net cloud radiative effect (W/m2)	−21.96 (−4.15)	−21.72 (−3.91)	−22.85 (−5.04)	−23.21 (−5.40)	−21.10 (−3.29)

Note. Parentheses show the biases against the observation.

Abbreviations: CNTL, control run; DOUB, double; FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple; QUAD, quadruple; TOA, top-of-atmosphere; 16XL, sexdecuple.

Discussion The Importance of Large-Scale Vertical Advection in FIVE

As mentioned in Section 2.3, the large-scale vertical advection computed in FIVE is necessary to balance entrainment via turbulence scheme. Figure 13 shows the comparison of FIVE_OCT and FIVE_OCT_noLS to quantify the impact of the large-scale vertical advection on the high-resolution VEP grid versus the standard 72-layer grid. Figure 13a shows that without the adjustment of vertical advection (i.e., remapping) in FIVE, the low-level cloud amount is reduced as much as 10%, especially in the marine stratocumulus regions. Corresponding differences are also found in the SWCRE (Figure 13b). Consistent with Y17, this test indeed shows that all four processes (i.e., microphysics, radiation, turbulence, and large-scale vertical advection) need to be applied on the VEP grid for a reasonable match with the benchmark simulations and the observations.

Figure 13. (a) The differences of low-level cloud amount between FIVE_OCT_noLS and FIVE_OCT. (b) The differences of shortwave cloud radiative effect between FIVE_OCT_noLS and FIVE_OCT. The results are 2 years average. FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple.

CFL Condition in E3SM-FIVE

As previously mentioned, no time step reduction is required when running E3SM-FIVE at LES-like vertical resolutions. This is a substantial performance advantage of E3SM-FIVE and is counter to B21, in which their high vertical resolution benchmark simulations were subject to time stepping constraints. This brings into question the Courant-Friedrichs-Lewy (CFL) condition for stable numerical integration of partial differential equations. Normally, the CFL condition should be considered for explicit time integration schemes to set an appropriate time step size. E3SM-FIVE is not constrained by the CFL condition because most of the physics schemes selected for VEP use numerically stable schemes for large time step sizes.

The CLUBB and MG2 parameterizations use an implicit scheme and time-split sedimentation scheme, respectively. The vertical advection uses a semi-Lagrangian scheme, so it is not subject to stringent time step limitation either. However, it is important to point out that neither the semi-Lagrangian algorithm performed in the vertical advection, nor the implicit treatment in CLUBB are a perfect solution to large time steps with thin layers. Even though the model remains stable this does not necessarily mean the solution is insensitive when large time step sizes are used. For instance, it would be possible to get negative air mass within thin model layers with large time steps. While we explored the possibility of this with idealized SCM studies (Bogenschutz et al., 2020) and found no evidence of this occurring for a marine stratocumulus case with high vertical resolution and long time steps, we note that a deeper examination of potential sensitivities should be explored.

Finally, we mention that the time step constraint in the benchmark experiments is associated with the ZM deep convection scheme, which has been tested in a sensitivity simulation in B21. They found that their OCT simulation ran stably with ZM shut off, with CLUBB acting as a deep convection parameterization, with default model time steps.

Time Step Sensitivity Test

Previous studies have demonstrated that climate variables in GCMs are sensitive to model time step, especially those associated with deep and shallow convective parameterization (Williamson, 2013; Yu & Pritchard, 2015). Williamson (2013) suggested that many of these sensitivities may be due to convective parameterization schemes failing to effectively adjust moist instability by vertical redistribution and associated condensation when the adjustment timescales assumed in convective parameterizations are longer than a GCM time step. B21 also demonstrated that their high-resolution benchmark simulations were sensitive to time step settings and they concluded that these sensitivities likely arise from the ZM deep convection scheme.

We conducted an additional test to determine if CLUBB and MG2 have a time step sensitivity at high vertical resolution. Figures 14a and 14b show that the differences of low-level cloud amount between FIVE_OCT_t150 (in which CLUBB and microphysics time steps were reduced from 300 to 150 s) and FIVE_OCT are largely negligible. While there exist some minor differences of SWCRE between FIVE_OCT_t150 and FIVE_OCT in Southeast Asia, this experiment does not show a significant sensitivity in the low-cloud regions we focused on. Overall, reducing time step in CLUBB and microphysics schemes does not appear to substantially affect the long-term climate trend nor the climatological stratocumulus results.

Figure 14. (a) The differences of low-level cloud amount between FIVE_OCT_t150 and FIVE_OCT. (b) The differences of shortwave cloud radiative effect between FIVE_OCT_t150 and FIVE_OCT. (c and d) are the same as (a and b), respectively, but the differences between FIVE_OCT_d900 and FIVE_OCT. The results are 2 years average. FIVE, Framework for Improvement by Vertical Enhancement; OCT, octuple.

The results for the FIVE_OCT_d900 experiment (where E3SM time step was reduced from 1,800 s to 900 s) also provides similar conclusions. Figure 14c shows the differences of low-level cloud amount between FIVE_OCT_d900 and FIVE_OCT. We reduced the time step of E3SM dynamics and physics by half, but the low-level cloud amount provides no significant changes compared to FIVE_OCT, only small increases in the ITCZ. While minor differences of SWCRE exist between FIVE_OCT_d900 and FIVE_OCT, there does not appear to be any significant systematic sensitivity (Figure 14d). The RMSE scores of precipitation for FIVE_OCT_t150 and FIVE_OCT_d900 are 1.21 and 1.23 mm day⁻¹, respectively, computed relative to the observations. Compared to the RMSE in FIVE_OCT (1.14 mm day⁻¹), the errors in FIVE_OCT_t150 and FIVE_OCT_d900 are slightly higher but not nearly as degraded as the RMSE in OCT (1.66 mm day⁻¹; Table 6). Overall, our results show that E3SM-FIVE is not sensitive to time step.

These results suggest that the large sensitivities seen in the tropics for the OCT simulation in B21 are related to sensitivities in the vertical resolution rather than the model time step, arising from the ZM deep convection scheme.

Future Applications of FIVE

Significant computational savings is one of the main benefits for using E3SM-FIVE. The total cost is less than the benchmark runs, especially at LES-like high vertical resolutions where we see substantial improvements in the simulation of marine stratocumulus. However, costs quickly mount when the number of VEP levels increases, even if no time step decrease is required (Table 4). The current version of E3SM-FIVE uses a common fixed VEP grid for all columns. Since the cost associated with FIVE is tightly related to the number of VEP levels, we expect that the VEP cost burden can be reduced by applying a variant of the adaptive vertical grid (AVG) method (Marchand & Ackerman, 2011) to the VEP grid. The application of an AVG scheme in E3SM-FIVE is an on-going project, which aims to allow the vertical extent of the high resolution region and the number of vertical levels of the VEP grid in each column to dynamically adapt as the solution evolves.

In spite of the improvements that our simulations demonstrate, our current highest vertical resolution results still show less stratocumulus in the coastal regions of California and Peru, compared to the observations, and we hypothesize that these deficiencies probably require concurrent increases in horizontal and vertical resolution. One potential application of FIVE is to use horizontal regional refinement over stratocumulus regions (Tang et al., 2019). Using this method would allow us to have concurrent horizontal and vertical resolution increases, but only in the regions where they are desired to mitigate excessive computational cost.

FIVE could also be applied to super-parameterized (SP) GCMs (Grabowski, 2001; M. Khairoutdinov et al., 2005; M. F. Khairoutdinov & Randall, 2001; Randall et al., 2003) with the idea that the embedded cloud resolving model (CRM) would be run at a higher vertical resolution while keeping the host GCM at the standard vertical resolution. It would also be possible to apply FIVE to select physics schemes within the CRM itself, while keeping the CRM at the standard GCM resolution, to further mitigate the computational expense. SP tries to address shortcomings of conventional GCMs by embedding a small domain cloud-resolving models in each global model grid column. Marchand and Ackerman (2010) investigated the cloud cover in a 1 km horizontal grid resolution of an embedded cloud system resolving model used in SP GCMs and the results show higher horizontal resolution actually decreased low cloud cover. However, increasing vertical resolution with higher horizontal resolution helped to restore low-cloud cover and modestly improved cloud top height.

Typical SP implementations have used CRMs with 1–4 km horizontal resolution and a coarse vertical resolution encompassing 30–50 vertical layers, which have not been able to resolve the turbulent eddies that form low cloud due to grid resolution limitations. While this has produced promising effects for deep convection, it is known that accurate representation of cloud-top-entrainment plays a crucial role in the realistic simulation of low clouds. This requires extremely fine vertical grid spacing (5–25 m) and horizontal grid spacing (5–100 m) in the embedded model (Grabowski, 2016). Thus, applying FIVE in SP can serve the purpose of finer vertical resolution in the CRM to accurately simulate turbulence and entrainment processes near sharp temperature inversions, but retaining the relatively coarse vertical resolution for the host model to reduce computational cost.

Summary

The aim of this work is to implement a new computational method, the FIVE, into the E3SM. Three physics schemes, the CLUBB turbulence scheme, the MG2 microphysics scheme, and the RRTMG radiation schemes as well as vertical advection, are interfaced to VEP, which allows for these schemes to be computed on a higher vertical resolution grid compared to the rest of the E3SM model. This is the first time, to our knowledge, that such a framework has been applied to a GCM. For our proof of concept implementation, we focus on the climatological effects of subtropical marine stratocumulus, since this is a regime that is known to be sensitive to vertical resolution.

The three physics schemes we interfaced in E3SM-FIVE are essential to be run at high vertical resolution for a successful simulation of stratocumulus, owing to the tight interaction between turbulence, microphysics, and radiation. In addition to these three physics schemes, interfacing FIVE to the computation of large-scale vertical advection in the dynamical core is necessary to balance entrainment via the turbulence scheme. Our sensitivity study shows it helps to increase the low cloud amount by ∼10% in the marine stratocumulus regions, as well as helping to ameliorate radiational biases.

In this study, we used VEP for turbulence, microphysics, radiation parameterizations, and vertical advection, and demonstrated the better representation of subtropical boundary layer clouds. The configuration of the CNTL is based on the configuration of E3SMv1 (72 vertical layers). Four principle simulations were designed to double (FIVE_DOUB; 92 vertical layers), quadruple (FIVE_QUAD; 132 vertical layers), octuple (FIVE_OCT; 212 vertical layers) and sexdecuple (FIVE_16XL; 372 vertical layers) the vertical resolution between 995 and 700 hPa. The purpose of the experimental design is to determine if the representation of marine stratocumulus is improved when the high vertical resolution is applied to select physical processes.

Our results show when the vertical resolution approaches LES-like resolutions in FIVE_OCT and FIVE_16XL, the low cloud amount increases by more than 30% in the southeastern Pacific Ocean and the improvement seems to converge at these scales. The SWCRE is also improved, mostly in the southeast Pacific Ocean. Our simulations show that the improvement of low-level cloud bias focuses on the offshore “core” regions but not along the coasts. The improvement of the low-level cloud bias along the coasts becomes visible only in FIVE_16XL. It is unclear if further vertical refinement would lead to further decreases in biases in these regions, but we speculate that concurrent increases in horizontal and vertical resolution are needed to significantly ameliorate coastal stratocumulus biases.

Compared to the E3SM benchmarks, E3SM-FIVE limits additional computational cost from the increased number of levels, especially when running at LES-like vertical resolutions. No reduction of E3SM time step is required with any of the E3SM-FIVE configurations, compared to the E3SM benchmark runs, which is partially why E3SM-FIVE greatly reduces computational cost compared with high vertical resolution simulations without FIVE. The time step constraint in the benchmark simulations is likely associated with the ZM deep convection scheme, which has been tested in a benchmark sensitivity simulation (see Bogenschutz et al., 2021). In addition, the ZM deep convection scheme appears to be sensitive to higher vertical resolution, and it results in degrading skill scores of precipitation and clouds in the deep convective tropics as the vertical resolution becomes very fine. E3SM-FIVE does not compute ZM deep convective processes on the VEP grid and in our sensitivity experiments, E3SM-FIVE is not sensitive to time step and does not suffer from stringent time step limitations. In other words, in E3SM-FIVE, we can avoid negative consequences of running parameterizations which may be negatively impacted by higher vertical resolution.

Regarding future applications of FIVE, we discussed an ongoing project for AVG for VEP, for cost mitigation, which allows the vertical extent of the high-resolution region and the number of vertical levels of the VEP grid in each column to dynamically adapt as the solution evolves. Using FIVE with horizontal mesh refinement is one potential application to concurrently increase the horizontal and vertical resolution over stratocumulus regions. We hypothesize this is necessary to ameliorate stubborn coastal stratocumulus biases, which do not appear to be greatly improved by vertical resolution increases alone. Another application of FIVE is in regard to the embedded CRMs in super-parameterization, where the idea is to increase the vertical resolution of the embedded CRM, or the CRM physics, but not of the host model.

Finally, although this study focuses on the marine stratocumulus regime for our proof of concept implementation, the application of FIVE in E3SM is not limited to the lower troposphere. For example, one could increase the vertical resolution in VEP to the upper troposphere to examine the effects of vertical resolution on cirrus clouds.

Acknowledgments

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research and Office of Biological and Environmental Research, Scientific Discovery through Advanced Computing (SciDAC) program under Award Number DE-SC0018650. Work at LLNL was performed under the auspices of the U.S. DOE by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. LLNL IM: LLNL-JRNL-810691.

Data Availability Statement

The model code used in this study is located at https://doi.org/10.5281/zenodo.3893210. The output from the E3SM-FIVE simulations can be found at https://doi.org/10.5281/zenodo.3887276. All simulations use the FC5AV1C-L compset for EAMv1.