1.2. Technical Overview
1.2.1. Prerequisites
1.2.1.1. Minimum System Requirements
UFS applications, models, and components require a UNIX-based operating system (i.e., Linux or MacOS).
Additionally, users will need:
Disk space: ~130 GB
~19 GB for basic land-only case (2.4 GB for Land DA System, 136 KB for experiment directory, 16 GB for staging and output)
111 GB for full set of Land DA data (can use smaller subset if needed)
26 CPU cores (13 CPUs may be possible, but it has not been tested)
1.2.1.2. Software Prerequisites
The Land DA System requires:
An MPI implementation
A Fortran compiler
Python
Lmod
spack-stack (v1.9.2)
These software prerequisites are pre-installed in the Land DA container and on other Level 1 systems (see below for details). However, users on non-Level 1 systems will need to install them.
Before using the Land DA container, users will need to install Singularity/Apptainer and an Intel MPI (available free here).
1.2.2. Supported Systems for Running Land DA
Four levels of support have been defined for UFS applications, and the Land DA System operates under this paradigm:
Level 1 (Preconfigured): Prerequisite software libraries are pre-built and available in a central location; code builds; full testing of model.
Level 2 (Configurable): Prerequisite libraries are not available in a centralized location but are expected to install successfully; code builds; full testing of model.
Level 3 (Limited-test platforms): Libraries and code build on these systems, but there is limited testing of the model.
Level 4 (Build-only platforms): Libraries and code build, but running the model is not tested.
1.2.2.1. Level 1 Systems
Preconfigured (Level 1) systems for Land DA already have the required external libraries available in a central location via spack-stack and the jedi-bundle. Land DA is expected to build and run out-of-the-box on these systems, and users can download the Land DA code without first installing prerequisite software. With the exception of the Land DA container, users must have access to these Level 1 systems in order to use them. For the most updated information on stack locations, compilers, and MPI, users can check the run version file for their machine of choice.
Platform |
Compiler |
MPI |
spack-stack Installation |
jedi-bundle Installation |
|---|---|---|---|---|
Gaea-C6 |
intel/2023.2.0 |
mpich/8.1.30 |
/ncrc/proj/epic/spack-stack/c6/spack-stack-1.9.2/envs/ue-intel-2023.2.0/install/modulefiles/Core /ncrc/proj/epic/spack-stack/c6/modulefiles |
/gpfs/f6/bil-fire8/world-shared/UFS_Land-DA_v3.0/jedi_bundle_sync |
Hercules |
intel/2024.2.1 |
impi/2021.13 |
/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/modulefiles |
/work/noaa/epic/UFS_Land-DA_v3.0/jedi_bundle_hercules |
Orion |
intel/2024.2.1 |
impi/2021.13 |
/apps/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.1.0/install/modulefiles |
/work/noaa/epic/UFS_Land-DA_v3.0/jedi_bundle_orion |
Ursa |
intel/2024.2.1 |
impi/2021.13 |
/contrib/spack-stack/spack-stack-1.9.2/envs/ue-oneapi-2024.2.1/install/modulefiles |
/scratch3/NAGAPE/epic/UFS_Land-DA_v3.0/jedi_bundle_ursa |
Container |
intel-oneapi-compilers/2021.10.0 |
intel-oneapi-mpi/2021.9.0 |
/opt/spack-stack/spack-stack-1.6.0/envs/fms-2024.01/install/modulefiles/Core (inside the container) |
/opt/jedi-bundle (inside the container) |
1.2.2.2. Level 2-4 Systems
On non-Level 1 platforms, the Land DA System can be run within a container that includes the prerequisite software; otherwise, the required libraries will need to be installed before building the Land DA System. Once these prerequisite libraries are installed, Land DA should build and run successfully. However, users may need to perform additional troubleshooting on Level 3 or 4 systems since little or no pre-release testing has been conducted on these systems. Currently, the Land DA System is not supported on Level 2-4 systems except via container.
1.2.3. Code Repositories and Directory Structure
1.2.3.1. Hierarchical Repository Structure
The main repository for the Land DA System is named land-DA_workflow;
it is available on GitHub at https://github.com/ufs-community/land-DA_workflow.
This umbrella repository uses Git submodules and an app_build.sh file to pull in code from the appropriate versions of external repositories associated with the Land DA System. Table 1.2 describes the various submodules that form the UFS Land DA System.
Land DA Submodule Name |
Repository Name |
Repository Description |
Authoritative Repository URL |
|---|---|---|---|
apply_incr.fd |
land-apply_jedi_incr |
Contains code that applies the JEDI-generated DA increment to UFS |
|
calcfIMS.fd |
land-SCF_proc |
Code for processing IMS input ASCII files on the UFS model grid |
|
jcb-algorithms |
jcb-algorithms |
Contains YAML algorithm files (e.g., LETKF-OI, 3DVar) for the JEDI Configuration Builder; these files contain the high-level configuration structure that is prescribed by the JEDI data assimilation system. |
|
jcb-gdas |
jcb-gdas |
Contains information for different types of analysis (e.g., snow, marine, atmosphere) |
|
ufs_model.fd |
ufs-weather-model |
Repository for the UFS Weather Model (WM). Contains a number of subcomponents, including the Noah-MP land surface model, the FV3 dynamical core, and CDEPS. |
|
UFS_UTILS.fd |
UFS_UTILS |
Repository containing UFS Utilities |
Note
The prerequisite libraries (including NCEP Libraries and external libraries) are not included in the UFS Land DA System repository. The spack-stack repository assembles these prerequisite libraries. Spack-stack has already been built on preconfigured (Level 1) platforms. However, it must be built on other systems. See the spack-stack Documentation for details on installing spack-stack.
1.2.3.2. File & Directory Structure
The land-DA_workflow is evolving to follow the NCEP Central Operations (NCO) WCOSS Implementation Standards. When the land-DA_workflow repository is cloned with the --recursive argument, the specific GitHub repositories described in /sorc/app_build.sh are cloned into sorc. The diagram below illustrates the file and directory structure of the Land DA System. Directories in parentheses () are only visible after the build step. Some files and directories have been removed for brevity.
land-DA_workflow
├── doc
├── (exec)
├── fix
├── jobs
| └── JLANDDA_*
├── (lib64)
├── modulefiles
│ ├── conda.lua
│ ├── ufs_common.lua
│ ├── ufsland_<platform>.<compiler>.lua
│ └── wflow_<platform>.lua
├── parm
│ ├── config_samples
│ │ └── config.*.yaml
│ ├── jedi
│ │ ├── jcb-algorithms
│ │ ├── jcb-base_snow.yaml.j2
│ │ └── jcb-gdas
│ ├── templates
│ │ ├── template.ATML.*
│ │ ├── template.LND.*
│ │ └── template.land_analysis.yaml
│ ├── automate_launch_script.py
│ ├── detect_platform.sh
│ ├── run_container_executable.sh
│ └── setup_wflow_env.py
├── scripts
| └── exlandda_*.sh
├── sorc
| ├── UFS_UTILS.fd
│ ├── CMakeLists.txt
| ├── app_build.sh
| ├── apply_incr.fd
| | └── sorc
| | ├── apply_incr_noahmp_snow.f90
| | └── NoahMPdisag_module.f90
│ ├── (build)
| ├── calcfIMS.fd
| | └── sorc
| | ├── IMSaggregate_mod.f90
| | └── driver_fIMS.f90
│ ├── jcb-algorithms
| ├── jcb-gdas
│ ├── test
│ │ ├── <machine>_ctest.sh
│ │ ├── run_<machine>_ctest.sh
│ │ └── run_ctest_platform.sh
│ ├── tile2tile_converter.fd
│ └── ufs_model.fd
| ├── CDEPS-interface
| ├── FV3
| └── NOAHMP-interface
├── ush
| ├── *_2ioda.py
| ├── compare.py
| ├── fill_jinja_template.py
| ├── hofx_analysis_stats.py
| ├── jcb_setup.py
| ├── letkf_create_ens.py
| └── plot_*.py
├── versions
├── LICENSE
└── README.md
Table 1.3 describes the organizational structure of the Land DA System. Section 1.2.3.1 describes the Land DA System submodules. Users may reference the NCO Implementation Standards (p. 19) for additional details on repository structure in NCO-compliant repositories.
Directory Name |
Description |
|---|---|
doc |
Repository documentation |
exec |
Binary executables |
fix |
Location of fix/static files |
jobs |
|
lib64 |
Model-specific libraries |
modulefiles |
Files that load the modules required for building and running the workflow |
parm |
Parameter files used to configure the model, physics, workflow, and various components |
scripts |
Scripts launched by the J-jobs |
sorc |
External source code used to build the Land DA System |
ush |
Utility scripts |
versions |
Contains |