An Architecture for Many-Component

Regional Hydrologic Modeling

Cecelia DeLuca
NOAA Environmental Software Infrastructure and Interoperability (NESII)
NOAA ESRL/University of Colorado
Collaborating team: Rocky Dunlap/CIRES, Dan Rosen/CIRES,
Scott Peckham/University of Colorado, Wei Yu/NCAR, Dave Gochis/NCAR,
Sujay Kumar/NASA, Mike Ek/NOAA EMC, Jiariu Dong/NOAA EMC
3rd CUAHSI Conference on Hydroinformatics
July 17, 2015

Bridging Disciplines and Approaches

The climate and (to a lesser extent) weather communities have

engaged in many-component (5-50 component) modeling efforts
for more than a decade

These efforts established practices and have evolved tools that

could be leveraged or extended by the hydrologic community

Modeling efforts at the National Water Center have the potential to

advance the representation of hydrology in coupled modeling
systems used for a variety of purposes

Motivates efforts to bridge disciplines and experiment with

connections of modeling frameworks with support from the NSF
Earth System Bridge project

Federal Coupled Modeling System Examples

NEMS: NOAA Environmental Modeling
System

Next-generation operational prediction for

weather through seasonal time scales
Some applications in operations now, some in
development

NASA GEOS-5 Global Circulation Model and

ModelE

Research in data assimilation techniques and

utilization of satellite measurements
Seasonal forecasting, climate forecasting, creation
of reanalysis data sets

CESM: Community Earth System Model

Navy Forecast Systems (COAMPS, NavGEM)

Research into all aspects

of the climate system
National and international
assessments, including
participation in the
Intergovernmental Panel
on Climate Change
assessment reports

Research and operational weather forecasting in

support of military operations and national
security

Surface winds from

COAMPS Navy model

Earth System Prediction Suite

ESPS COUPLED MODELING SYSTEMS
Model
Driver

NEMS
and CFS

COAMPS

NavGEM

GEOS-5

ModelE

CESM

ATMOSPHERE MODELS
GSM

NMMB

The Earth System Prediction Suite is a

collection of federal and community
models and components that use the
Earth System Modeling Framework
(ESMF) with conventions called the
National Unified Operational Prediction
Capability (NUOPC) Layer

ESMF standard component interfaces

enable major U.S. centers to assemble
systems with components from different
organizations, and test a variety of
components more easily.

The multi-agency Earth System

Prediction Capability (ESPC) supports
adoption efforts

CAM
FIM

GEOS-5
Atmosphere
ModelE
Atmosphere
COAMPS
Atmosphere
NavGEM

NEPTUNE

OCEAN MODELS
MOM5

HYCOM

NCOM

POP

POM

SEA ICE MODELS

CICE

KISS

OCEAN WAVE MODELS

WW3

SWAN

LEGEND

Components are NUOPC compliant and the technical correctness of data

transfers in a coupled system has been validated.

Components and coupled systems are partially NUOPC compliant.

From Theurich et al. 2015,

in submission

Earth System Modeling Framework

The Earth System Modeling Framework
(ESMF) was initiated in 2002 as a multiagency response to calls for common
modeling infrastructure.
ESMF provides:
high performance utilities, including
grid remapping, data communications,
and model time management
an architecture for model construction
ESMF has become a standard for federal
research and operational models in
climate, weather, and space weather.

Metrics:
~6000 downloads
~100 components in use
~3000 individuals on info
mailing list
~40 platform/compilers
regression tested nightly
~6500 regression tests
~1M SLOC

https://www.earthsystemcog.org/projects/esmf/

ESMF Standard Component Interfaces

APIs in Fortran, C, and Python

All ESMF components have the same three standard

methods (these can have multiple phases)
-

Initialize
Run
Finalize

Each standard method has the same simple interface:

call ESMF_GridCompRun (myComp, importState, exportState,
clock, )
Where:
myComp points to the component
importState is a structure containing input fields
exportState is a structure containing output fields
clock contains timestepping information

Interfaces are wrappers and can often be introduced in a nonintrusive and high performance way, i.e. ESMF is designed to coexist
with native model infrastructure

ESMF Grid Remapping

Uniquely fast, reliable, and general interpolation weights computed in parallel in 3D space
Supported grids:
Logically rectangular and unstructured grids
Global and regional grids
2D and 3D grids
GIS formats through link with OpenClimateGIS (more on this)
Supported interpolation methods:
Nearest neighbor, higher order patch recovery, bilinear and 1st order conservative methods
Options for straight or great circle lines, masking, and a variety of pole treatments
Multiple ways to call ESMF grid remapping:
Generate and apply weights using the ESMF API, within a model
Generate and apply weights using ESMPy, through a Python interface
Generate weights from grid files using ESMF_RegridWeightGen, a command-line utility

Some
supported
grids
HOMME Cubed Sphere Grid with Pentagons
Courtesy Mark Taylor of Sandia

FIM Unstructured Grid

Regional Grid

OpenClimateGIS

OpenClimateGIS (OCGIS) is a Python-based, open source software library

enabling dynamic access to and manipulation of climate data

Its goal is to overcome barriers of usability of climate projections in adaptation

planning and resource management

Translate out of climate data formats to GIS-friendly formats

Select geographical regions of interest
Select times/levels of interest
Compute application-relevant indices
Convert to end-user and analysis-ready formats
Provide comprehensive metadata

Recently integrated with ESMPy for easier grid remapping from catchment
geometry to model grid

http://www.earthsystemcog.org/projects/openclimategis/

ESMPy + OpenClimateGIS

NFIE- inspired example

Subset high resolution climate precipitation data to local scale and then regrid to
hydrologic sub-basins

Source data: CF formatted precipitation data

file for the continental United States
(nldas_met_update.obs.daily.pr.1990.nc)

Output: Multi-dimensional precip values

(including time) on a subset of 3 catchment
basins in region of interest after generation
and application of conservative interpolation
weights

NFIE Example Workflow

Using ESMPy and OpenClimateGIS a Python workflow was created to
subset and conservatively regrid output from daily precipitation data
(NetCDF) and a hydrologic sub-basin shapefile
Output shapefile contains the full time series of regridded precipitation
data and all original attributes (i.e. ComID)

More information on the NFIE workflow example is available here:

http://bit.ly/1Hqj3sy

NUOPC Interoperability Layer

The National Unified Operational Prediction Capability
(NUOPC) is a consortium of operational weather
prediction centers and their research partners
The NUOPC Layer adds to ESMF:

Definitions for model component interactions during

Initialize, Run, Finalize (I/R/F) for interoperability even with
complex initialization interactions
Extensible, generic components that implement standard
phases of the I/R/F
A field dictionary that supports multiple conventions and
aliases, used for matching fields between components
Component Explorer and Compliance Checker tools that
report component incompatibilities detected during runtime
Static compliance checking and code generation via the
Cupid Integrated Development Environment (IDE)

https://www.earthsystemcog.org/projects/nuopc/

NUOPC API
calls and supplements

ESMF API

The National Unified Operational

Prediction Capability
NUOPC introduces a set of pre-fabricated model component templates for
building coupled systems
NUOPC Generic Components
Driver

Connectors

Model

Mediator

Simple driver (a) and

schematic of Navy regional
model COAMPS (b)

NUOPC wrappers or caps contain translations of native data structures (e.g.

grids, field data, time quantities) into ESMF data structures.

Cupid Development and Training Environment

NASA-funded, Eclipse plug-in Cupid parses NUOPC-based code and outlines

coupled model structure
Shows NUOPC compliance issues and generates code templates to satisfy
NUOPC compliance

NOAA Environmental Modeling System

The NOAA Environmental Modeling System (NEMS) is a rapidly evolving

coupled modeling system that will be the basis of critical predictive
systems at NOAA, including 10 day, six week, and 9 month forecasts

Global and regional applications

Components will include

-

Atmosphere
Ocean
Land
Wave
Sea ice
Coastal/storm surge
Hydraulics
Ionosphere
Aerosol/chemistry

https://www.earthsystemcog.org/projects/nuopc/

NEMS Status Current

NUOPC Driver
Model:
OCN
IPM

Model:
ROF
CST

Coastal
ADCIRC

Mediator

Model:
ROF
HYD

Hydraulics
WRF-Hydro

Model:
Model:
WAV
WAV

Wave
WaveWatch III

Model:
Model:
GLC
LND

Land
LIS

Model:
Model:
GLC
CHM

Aerosol/Chem
GOCART
CMAQ

Atmosphere
GSM
NMM
as chosen

Model:
ATM

Ice
CICE
as chosen

Model:
ICE

Ocean
POP
MOM5
MOM6
HYCOM

Model:
OCN

Mediator
Mediator

Ionosphere
IPE

nems

Current status: white=coupled, grayed out=in progress, solid=not started

NEMS Planned 12/2015

NUOPC Driver
Model:
OCN
IPM

Model:
ROF
CST

Coastal
ADCIRC

Mediator

Model:
ROF
HYD

Hydraulics
WRF-Hydro

Model:
Model:
WAV
WAV

Wave
WaveWatch III

Model:
Model:
GLC
LND

Land
LIS

Model:
Model:
GLC
CHM

Aerosol/Chem
GOCART
CMAQ

Atmosphere
GSM
NMM
as chosen

Model:
ATM

Ice
CICE
as chosen

Model:
ICE

Ocean
POP
MOM5
MOM6
HYCOM

Model:
OCN

Mediator
Mediator

Ionosphere
IPE

nems

Anticipated end of calendar year 2015: white=coupled, grayed out=in progress,

solid=not started

NEMS Mediator

Mediator transforms and transfers data can be multiple or specialized

Multiple coupling periods, now includes slow (ocean) and fast

(atmosphere and ice) coupling periods

The NEMS mediator performs the following functions:

Connects fields whose standard names match

Accumulates and averages fields as necessary

Merges fields with a generic merge method that allows for

weighting

Performs custom coupling operations, along with unit

transformations

Performs interpolation (fluxes are mapped bilinearly, states

conservatively, higher order also available)

More information about the mediator:

http://cog-esgf.esrl.noaa.gov/projects/couplednems/mediator_design

Sample NEMS Configure File

################################
# NEMS Run Time Configuration File #
################################
# MED #
med_model:
nems
med_petlist_bounds:
60 65
#ATM#
atm_model:
gsm
atm_petlist_bounds:
Processor layout

0 31

# OCN #
ocn_model:
ocn_petlist_bounds:

mom5
32 55

# ICE #
ice_model:
ice_petlist_bounds:

cice
56 59

Processor layout

# Run Sequence #
runSeq::
@7200.0
OCN -> MED
MED MedPhase_slow
MED -> OCN
OCN
@3600.0
MED MedPhase_fast_before
MED -> ATM
MED -> ICE
ATM
ICE
ATM -> MED
ICE -> MED
MED MedPhase_fast_after
@
@
::
Colors show actions
performed by:

Connectors (->)

Mediator (MED)

Models

(@) indicates coupling

timesteps

Assembling NEMS Modeling Applications

NEMS AppBuilder:

Enables users to construct a specific, versioned modeling application

from a versioned set of model components and configuration files from
multiple locations.

Helps to ensure that changes made to the different applications are

coordinated as they get checked back into the NEMS repository.

Implemented using SVN externals, can be used with git repositories

The AppBuilder is based on low-level
terminal-based Unix utilities for
maximum ease of use and
portability. A command-line version
will be available shortly.

More about the AppBuilder:

http://cog-esgf.esrl.noaa.gov/projects/couplednems/appbuilder

Running NEMS Applications

Component sets (compsets):

A labeling system that distinguishes different run configurations for

many-component modeling applications.

Labels are associated with scripts that pull together all the files and
inputs needed to run the specified configurations.

standard runs can be set up easily and consistently

effective way to implement regression testing across a coupled

system with many possible combinations of components

Each modeling application is typically associated with multiple

compsets.

More about compsets:

http://cog-esgf.esrl.noaa.gov/projects/couplednems/compsets

Using Compsets
Running Compsets:
./NEMSCompsetRun [COMPSET_LIST_FILE]

Compset syntax:
caselabel_architecture_model1[%opt1[%opt2[...[%optN]]]]_model2[...]
_..._modelN[...]

Where the file has the format:

### List of compsets ###
########################
AMIP_sbys_gsm
2009_nems_gsm_cice_mom5
2011_sbys_gsm%wam
2011_sbys_gsm%wam%ndsl

!
!
!
!

stand-alone GSM - fake example

NEMS mediator coupled GSM-CICE-MOM5
stand-alone GSM run as WAM
stand-alone GSM run as WAM with NDSL

Supported compsets:
http://cog-esgf.esrl.noaa.gov/projects/couplednems/supported_compsets

NEMS Seasonal 0.1 Milestone

A first version (0.1) of the NEMS global seasonal prediction was completed in
June 2015.

Three-way coupled atmosphere-ocean-ice system - GSM (T126) -MOM5 (1

deg) CICE (1 deg)

Image courtesy of Fei Liu, NOAA CIRES

SST after 5 days of a 15 day run

Focus so far has been on technical correctness
Model initialization is not fully in place
Next steps focus on adding grid resolutions and initializing from CFSR so
comparative runs with last operational model can begin

Next Milestones
August 2015 NEMS regional 0.1 milestone

WRF-Hydro hydrology and Noah-MP (via the NASA Land Information

System) running side by side uncoupled with other active NEMS
components (still global atmosphere and ocean, CICE ice model)

Important to have ability to validate each component standalone!

Currently WRF-Hydro is capped with NUOPC and running in NEMS (with

a minor I/O glitch) and LIS is in the process of being capped

December 2015 NEMS regional 0.2 milestone

WRF-Hydro hydrology and Noah-MP or Noah (via the NASA Land

Information System) running coupled with other NEMS components
(target regional NMMB atmosphere and regional HYCOM ocean, CICE
ice model)

May 2016 NEMS regional 0.3 milestone

Flexible, but likely ADCIRC added to coupled system

Potential for experimental connection to CSDMS components

Conclusion

A prototype many-component system is being constructed at NOAA

that includes a whole Earth set of coupled components, using
ESMF/NUOPC Layer infrastructure

An initial delivery of a seasonal application shows reasonable behavior

and is entering an evaluation phase

Next NEMS milestones include a regional application, and bring in WRFHydro, a separate land component, and a coastal/storm surge model

This regional system can serve as an exploratory testbed for both

technical and scientific aspects of this complex coupling, and a bridge
to connect hydrologic and climate/weather communities

Thank you!
Any questions?