2.1. Land DA Workflow (Hera & Orion)

This chapter provides instructions for building and running basic Land DA cases for the Unified Forecast System (UFS) Land DA System. Users can choose between two options:

  • A Dec. 21, 2019 00z sample case using ERA5 data with the UFS Land Driver (settings_DA_cycle_era5)

  • A Jan. 3, 2000 00z sample case using GSWP3 data with the UFS Noah-MP land component (settings_DA_cycle_gswp3).

Attention

These steps are designed for use on Level 1 systems (i.e., Hera and Orion) and may require significant changes on other systems. It is recommended that users on other systems run the containerized version of Land DA. Users may reference Chapter 2.2: Containerized Land DA Workflow for instructions.

2.1.1. Create a Working Directory

Create a directory for the Land DA experiment ($LANDDAROOT):

mkdir /path/to/landda
cd /path/to/landda
export LANDDAROOT=`pwd`

where /path/to/landda is the path to the directory where the user plans to run Land DA experiments.

2.1.2. Get Data

Table 2.1 shows the locations of pre-staged data on NOAA RDHPCS (i.e., Hera and Orion).

Table 2.1 Level 1 RDHPCS Data

Platform

Data Location

Hera

/scratch2/NAGAPE/epic/UFS_Land-DA/inputs

Orion

/work/noaa/epic/UFS_Land-DA/inputs

Users can either set the LANDDA_INPUTS environment variable to the location of their system’s pre-staged data or use a soft link to the data. For example, on Hera, users may set:

export LANDDA_INPUTS=/scratch2/NAGAPE/epic/UFS_Land-DA/inputs

Alternatively, users can add a soft link to the data. For example, on Orion:

cd $LANDDAROOT
ln -fs /work/noaa/epic/UFS_Land-DA/inputs

Users who have difficulty accessing the data on Hera or Orion may download it according to the instructions in Section 2.2.3. Users with access to data for additional experiments may use the same process described above to point or link to that data by modifying the path to the data appropriately.

Users who are not using Land DA on Hera or Orion should view Chapter 2.2 for instructions on running the containerized version of Land DA. Section 2.2.3 explains options for downloading the sample data onto their system.

2.1.3. Get Code

Clone the Land DA repository. To clone the develop branch, run:

git clone -b develop --recursive https://github.com/ufs-community/land-DA_workflow.git

To clone the most recent release, run the same command with release/public-v1.2.0 in place of develop:

git clone -b release/public-v1.2.0 --recursive https://github.com/ufs-community/land-DA_workflow.git

2.1.4. Build the Land DA System

  1. Navigate to the workflow directory, and source the modulefiles.

    cd $LANDDAROOT/land-DA_workflow
module use modulefiles
module load landda_<machine>.intel

    where <machine> is either hera or orion.

  2. Create and navigate to a build directory.

    mkdir build
cd build

  3. Build the Land DA System.

    ecbuild ..
make -j4

    If the code successfully compiles, the console output should end with:

    [100%] Completed 'ufs-weather-model'
[100%] Built target ufs-weather-model

    Additionally, the build directory will contain several files and directories along with a bin subdirectory with four executables:

    • apply_incr.exe

    • ufsLand.exe

    • vector2tile_converter.exe

    • tile2tile_converter.exe

2.1.5. Configure the Experiment

The develop branch includes two scripts with default experiment settings:

  • settings_DA_cycle_era5 for running a Dec. 21, 2019 00z sample case with the UFS Land Driver.

  • settings_DA_cycle_gswp3 for running a Jan. 3, 2000 00z sample case with the UFS Noah-MP land component.

To configure an experiment:

  1. Navigate back to the land-DA_workflow directory and check that the account, queue, and partition are correct in submit_cycle.sh.

    cd ..
vi submit_cycle.sh

    If necessary, modify lines 3 and 4 to include the correct account and queue (qos) for the system. It may also be necessary to add the following line to the script to specify the partition:

    #SBATCH --partition=my_partition

    where my_partition is the name of the partition on the user’s system.

    When using the GSWP3 forcing option, users will need to update line 7 to say #SBATCH --cpus-per-task=4. Users can perform this change manually in a code editor or run:

    sed -i 's/--cpus-per-task=1/--cpus-per-task=4/g' submit_cycle.sh

  2. When using GSWP3 forcing option, users may also have to alter MACHINE_ID in line 8 of settings_DA_cycle_gswp3. The default value is hera, but orion is another option:

    export MACHINE_ID=orion

    Users running the ERA5 case do not need to make this change.

  3. Configure other elements of the experiment if desired. The settings_* files contain reasonable default values for running a Land DA experiment. Users who wish to run a more complex experiment may change the values in these files and the files they reference using information in Sections 3.1 & 3.2.

2.1.6. Run an Experiment

The Land DA System uses a script-based workflow that is launched using the do_submit_cycle.sh script. This script requires a settings_DA_cycle_* input file that details all the specifics of a given experiment. For example, to run the ERA5 case, users would run:

./do_submit_cycle.sh settings_DA_cycle_era5

Users can replace settings_DA_cycle_era5 with a different settings file to run a different default experiment. Regardless of the file selected, the system will output a message such as Submitted batch job ########, indicating that the job was successfully submitted. If all goes well, one full cycle will run with data assimilation (DA) and a forecast.

2.1.7. Check Progress

To check on the experiment status, users on a system with a Slurm job scheduler may run:

squeue -u $USER

To view progress, users can open the log* and err* files once they have been generated:

tail -f log* err*

Users will need to type Ctrl+C to exit the files. For examples of what the log and error files should look like in a successful experiment, reference ERA5 Experiment Logs or GSWP3 Experiment Logs below.

Attention

If the log file contains a NetCDF error (e.g., ModuleNotFoundError: No module named 'netCDF4'), run:

python -m pip install netCDF4

Then, resubmit the job (sbatch submit_cycle.sh).

Next, check for the background and analysis files in the test directory.

ls -l ../landda_expts/DA_<data_source>_test/mem000/restarts/<vector/tile>``

where:

  • <data_source> is either era5 or gswp3, and

  • <vector/tile> is either vector or tile depending on whether ERA5 or GSWP3 forcing data was used, respectively.

The experiment should generate several files.

2.1.7.1. ERA5 Experiment Logs

For the ERA5 experiment, the log* file for a successful experiment will a message like:

Creating: .//ufs_land_restart.2019-12-22_00-00-00.nc
Searching for forcing at time: 2019-12-22 01:00:00

The err* file for a successful experiment will end with something similar to:

+ THISDATE=2019122200
+ date_count=1
+ '[' 1 -lt 1 ']'
+ '[' 2019122200 -lt 2019122200 ']'

2.1.7.2. GSWP3 Experiment Logs

For the GSWP3 experiment, the log* file for a successful experiment will end with a list of resource statistics. For example:

Number of times filesystem performed OUTPUT          = 250544
Number of Voluntary Context Switches                 = 3252
Number of InVoluntary Context Switches               = 183
*****************END OF RESOURCE STATISTICS*************************

The err* file for a successful experiment will end with something similar to:

+ echo 'do_landDA: calling apply snow increment'
+ [[ '' =~ hera\.internal ]]
+ /apps/intel-2022.1.2/intel-2022.1.2/mpi/2021.5.1/bin/mpiexec -n 6 /path/to/land-DA_workflow/build/bin/apply_incr.exe /path/to/landda_expts/DA_GSWP3_test/DA/logs//apply_incr.log
+ [[ 0 != 0 ]]
+ '[' YES == YES ']'
+ '[' YES == YES ']'
+ cp /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile1.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile2.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile3.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile4.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile5.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile6.nc /path/to/landda_expts/DA_GSWP3_test/DA/jedi_incr/
+ [[ YES == \N\O ]]