The process consists of five steps:

1) Identify a Docker image with Nvidia GPU support: To take advantage of the power of GPUs for our research, we need a base Docker system that supports GPU. After attempting to customize docker images for Nvidia GPU support without success, we decided to focus on identifying and evaluating preconfigured NVidia-docker images (nvcr.io) that already include compatible NVHPC programming environments and the necessary CUDA driver support. This allows us to efficiently run our simulations and computations on Nvidia GPUs across multiple computational resources.

2) Customize a Docker image for ELM simulation: On the top of the base Docker system, we integrate the ELM code, including the Offline ELM Model Testbed (OLMT) for automatic configuration and simulation of ELM at the site level. We accomplished this by leveraging our previous experience in developing customized docker images from https://ngee-arctic.ornl.govthe NGEE-Arctic project. Through a multi-step procedure for creating docker images, we successfully generated a new docker image for site-based ELM simulation on CPU.

3) Integrate a FUT tool for the standalone uELM module creation: We have developed a software tool (named SPEL [7] [8] ) we developed to enable uELM code generation and optimization over GPUs using Compiler directives within a FUT framework [9] . We use SPEL functions to instrument code segment into the ELM source code to capture and save the input and output data streams from a target ELM functional unit from a reference ELM simulation on CPUs. Then we create standalone uELM functional unit modules, in which, SPEL constructs a unit test driver that handles the initialization and reading of input parameters and data streams, executes a target uELM functional unit module, and saves the output datastreams, and compares the output datastreams with the output streams from the reference solution to verify the correctness and robustness of the standalone uEML module.

4) Deploy SPEL for GPU-ready uELM code porting and optimization: SPEL also provides functions to facilitate uELM GPU code generation and optimization. Users have the flexibility to develop their own strategies for quickly porting the GPU code, efficiently utilizing GPU resources, and maximizing the performance of individual uELM functional units. Additionally, users can combine several individual uELM functional units to generate an aggregated functional unit for further testing via SPEL.

5) Conduct end-to-end uELM code development on GPUs: The unit test of uELM via SPEL creates a good foundation for end-to-end code development on GPUs with compiler directives. Here, we enumerate four prevalent issues encountered during the process using NVHPC packages: a) The function of passing the array index to subroutines is flawed. We need to restructure the routines associated with the index and now pass the array elements explicitly as arguments. b) The atomic operations (atomic directive) are inconsistent, even though they might function within the FUT environment. We need to make sure these atomic operations are over the primitive datatypes. c) The most common error encountered is a fatal error, which complains that the variable (be it local or global) is only partially present on the device. Given that we lack direct access to the underlying algorithms and libraries, we are compelled to explore various methods to circumvent this issue. and s) The CUDA Debugger (within NVHPC) does not perform as efficiently as its CPU counterparts and demands excessive GPU memory. Consequently, we have automated the generation of variable checking and verification functions (in FORTRAN) using Python scripts.

We have achieved a threefold speedup on a fully loaded computing node within Summit [5] . The code is integrated into our standard container images.

We have developed Docker images for various computing resources (machine type, compiler, and library) with exemplary uELM configurations and simulation cases. These images are managed via Dockerhub, enabling developers to establish their Dockerhub accounts and pull these images into their local computing systems. Developers can initiate the container as a model user, construct new cases using the E3SM’s Common Infrastructure for Modeling Earth (CIME) or OLMT, and initiate simulations effortlessly. They also have the option to modify the code to make it GPU-ready with NVHPC/OpenACC, using SPEL (e.g., verification functions). Users can choose to exit and halt the container or detach from it while allowing it to run. They can later reattach to the running container for further action. Users can also create new Docker images with code changes or uELM cases for subsequent developments.

The containers are designed for uELM development on CPU-only or hydrid CPU/GPU systems. Each container is incorporated with the complete E3SM source code, which includes CIME and a collection of external models. They also contain the necessary input data for exemplary small simulation cases, along with useful tools like OLTM and SPEL. Most importantly, the container is equipped with all the necessary software libraries and environments for uELM code development and simulation right out of the box.

Several important file directories of a standard uELM container are listed in Table 1 , including the source code, input data and output directories, and tools that are used for uELM simulation case creation and code development.

The E3SM project employs GitHub as its version control system. Our container does not have a source control system. Users can establish an E3SM local repository on their computing resources initially, then map a local git folder to the uELM container with the “-volume” or “-mount” option. To circumvent potential folder privilege issues, it is advisable to initiate the uELM container as a “root” user. Following code alterations, users may disengage from the container and commit the modifications to the local E3SM repository. Alternatively, users can initiate a separate shell session and execute git commands outside the container while it remains running.

uELM simulations may require a substantial input dataset and generate voluminous outputs. It is impractical to incorporate large inputs and outputs into the container. Instead, users can utilize the “-volume” or “-mount” options to mount local data directory with the uELM container. In addition, if all computational resources are located within the same network, it is possible to share the uELM input and output data directory through Network File System (NFS). (Refer to Figure 1 or Section 2 for more details). Figure 1 illustrates the major components of the portable software environment and its deployments on computational resources within ORNLs open research domain.

The creation of the uELM software environment does not require powerful computing systems and can be implemented with a diverse range of computers, including laptops and PCs. Docker images can be deployed across a multitude of high-performance workstations or computational resources.

We have established a federated uELM development environment at ORNL. It consists of several laptops and PCs (such as MacBook Pros and Dell PCs). We have also connected two powerful stations: a NVidia DGX-station (2 20-core Intel Xeons, 256 GB Memory, 4 16 GB Nvidia V100, and 2TB storage), one virtual machine (5-core Intel Xeon, 128 GB Memory, one 32 GB NVidia V100 GPU, 3 TB NFS storage) in ORNL open cloud managed with Red-hat OpenStack. The 3T NFS data space is also accessible via the DGX station and an HPC cluster in CADES at ORNL.

The standard container encompasses an exemplary case that employs observational forcing data to drive the uELM simulations at 42 FluxNet sites globally ( https://fluxnet.org ). It incorporates all the necessary input datasets (e.g. domain, forcing, and surface properties), along with batch scripts, for configuration, construction, and automatic launch of the simulation. Additionally, a data duplication function is integrated, enabling users to replicate the 42 datasets to generate larger simulations as required for testing the parallel computing performance and scalability of the uELM code over HPC resources. Specifically, users can use the “OLMT_docker_42fluxnetsites_example.sh” script to create three simulation cases under the “/output/cime_case_dirs”. They are ad-spinup for model initialization, transit run (from 1980 to present), and future projection. Each case contains files and tools for case setup, build, and submission scripts, as well as input namelists. These simulations are used as reference solutions for both SPEL FUT test (Section 3.3) and end2end uELM development on GPUs (Section 3.4).

Figure 2 illustrates the location of these 42 FluxNet Site (left) and sample

output (soil temperature in top 10 cm of soil (TSOI_10CM)) of the uELM simulations.

We have incorporated the SPEL toolkit in the standard uELM development container. The folder “/app/SPEL_OpenACC” houses Python utilities and FORTRAN templates, which are designed to generate standalone functional unit testing programs for individual uELM modules. These include read and write codes for the IO of uELM module testing, a verification code to compare outputs from unit testing modules, and a FUT driver. Additionally, it contains a folder named “SourceFiles” that includes the ELM Fortran source files and GPU-ready ELM test modules. Another folder, “scripts”, comprises SPEL Python scripts and a few FORTRAN modules to generate ELM test modules. The generated unit test modules are stored in the “unit-tests” folder. Further information on the SPEL tool can be found in [7] .

We have included a complete FUT case within the container to illustrate the process of developing a functional unit test for a specific uELM module (LakeTemperature). Technically, the user employs the UnitTestforELM.py to generate the LakeTemperature unit test code (FORTRAN), creates executables for both CPUs and GPUs, and subsequently uses a verification code to ensure that the GPU code yields identical bit4bit results as the CPU code.

The standard container is utilized for end-to-end uELM development with users’ development branches (e.g., the latest GPU-ready uELM code that has shown a threefold speed increase on a single Summit node). In this scenario, we check out the latest uELM development branch into a local repository and subsequently mount the local development branch into the standard uELM container by substituting the default uELM source code. We further create uELM cases, modify input data, and create new docker images (e.g., wangdl1108/uelm_dev:e2e uELM_dev in dockerhub).

We followed the E3SM development guide during the development and testing of uELM. Initially, we utilized the ELM master branch (comes with the container) to generate baseline solutions via a series of developer tests. Subsequently, we mounted local uELM development branch and the local “inputdata” into the docker (see subsection 2.3) and initiated the same developer tests on CPUs to establish new baselines (called uELM_ref). We then compare uELM_ref with the baselines derived from the master branch to ensure that the uELM code modifications were either “bit4bit” or “climate change-related”. The next step involved repeating the uELM developer testing on GPUs and confirming that the newly generated simulation result was equivalent to the uELM_ref on CPUs.

The software environment offers a unique platform for interactive, collaborative uELM code development. We have established a Virtual Machine (VM) on the ORNL OpenCloud, equipped with a static IP address. We have also implemented a unified user ID, “cloud”, and user access is managed by the Public key infrastructure (PKI) systems. This allows authorized users to access the VM from any approved computer using the “cloud” account. One user can initiate the uELM software environment (container), and others can connect (attach) to the running container. This enables everyone to interact and collaborate through their own computers to co-develop the uELM simultaneously. This setup proves highly beneficial for uELM development in a geographically distributed manner.