2.1. Land DA Workflow (Hera & Orion)
This chapter provides instructions for building and running basic Land DA cases for the Unified Forecast System (UFS) Land DA System. Users can choose between two options:
A Dec. 21, 2019 00z sample case using ERA5 data with the UFS Land Driver (
settings_DA_cycle_era5
)A Jan. 3, 2000 00z sample case using GSWP3 data with the UFS Noah-MP land component (
settings_DA_cycle_gswp3
).
Attention
These steps are designed for use on Level 1 systems (i.e., Hera and Orion) and may require significant changes on other systems. It is recommended that users on other systems run the containerized version of Land DA. Users may reference Chapter 2.2: Containerized Land DA Workflow for instructions.
2.1.1. Create a Working Directory
Create a directory for the Land DA experiment ($LANDDAROOT
):
mkdir /path/to/landda
cd /path/to/landda
export LANDDAROOT=`pwd`
where /path/to/landda
is the path to the directory where the user plans to run Land DA experiments.
2.1.2. Get Data
Table 2.1 shows the locations of pre-staged data on NOAA RDHPCS (i.e., Hera and Orion).
Platform |
Data Location |
---|---|
Hera |
/scratch2/NAGAPE/epic/UFS_Land-DA/inputs |
Orion |
/work/noaa/epic/UFS_Land-DA/inputs |
Users can either set the LANDDA_INPUTS
environment variable to the location of their system’s pre-staged data or use a soft link to the data. For example, on Hera, users may set:
export LANDDA_INPUTS=/scratch2/NAGAPE/epic/UFS_Land-DA/inputs
Alternatively, users can add a soft link to the data. For example, on Orion:
cd $LANDDAROOT
ln -fs /work/noaa/epic/UFS_Land-DA/inputs
Users who have difficulty accessing the data on Hera or Orion may download it according to the instructions in Section 2.2.3. Users with access to data for additional experiments may use the same process described above to point or link to that data by modifying the path to the data appropriately.
Users who are not using Land DA on Hera or Orion should view Chapter 2.2 for instructions on running the containerized version of Land DA. Section 2.2.3 explains options for downloading the sample data onto their system.
2.1.3. Get Code
Clone the Land DA repository. To clone the develop
branch, run:
git clone -b develop --recursive https://github.com/ufs-community/land-DA_workflow.git
To clone the most recent release, run the same command with release/public-v1.2.0
in place of develop
:
git clone -b release/public-v1.2.0 --recursive https://github.com/ufs-community/land-DA_workflow.git
2.1.4. Build the Land DA System
Navigate to the workflow directory, and source the modulefiles.
cd $LANDDAROOT/land-DA_workflow module use modulefiles module load landda_<machine>.intel
where
<machine>
is eitherhera
ororion
.Create and navigate to a
build
directory.mkdir build cd build
Build the Land DA System.
ecbuild .. make -j4
If the code successfully compiles, the console output should end with:
[100%] Completed 'ufs-weather-model' [100%] Built target ufs-weather-model
Additionally, the
build
directory will contain several files and directories along with abin
subdirectory with four executables:apply_incr.exe
ufsLand.exe
vector2tile_converter.exe
tile2tile_converter.exe
2.1.5. Configure the Experiment
The develop
branch includes two scripts with default experiment settings:
settings_DA_cycle_era5
for running a Dec. 21, 2019 00z sample case with the UFS Land Driver.
settings_DA_cycle_gswp3
for running a Jan. 3, 2000 00z sample case with the UFS Noah-MP land component.
To configure an experiment:
Navigate back to the
land-DA_workflow
directory and check that the account, queue, and partition are correct insubmit_cycle.sh
.cd .. vi submit_cycle.sh
If necessary, modify lines 3 and 4 to include the correct account and queue (qos) for the system. It may also be necessary to add the following line to the script to specify the partition:
#SBATCH --partition=my_partition
where
my_partition
is the name of the partition on the user’s system.When using the GSWP3 forcing option, users will need to update line 7 to say
#SBATCH --cpus-per-task=4
. Users can perform this change manually in a code editor or run:sed -i 's/--cpus-per-task=1/--cpus-per-task=4/g' submit_cycle.sh
When using GSWP3 forcing option, users may also have to alter
MACHINE_ID
in line 8 ofsettings_DA_cycle_gswp3
. The default value ishera
, butorion
is another option:export MACHINE_ID=orion
Users running the ERA5 case do not need to make this change.
Configure other elements of the experiment if desired. The
settings_*
files contain reasonable default values for running a Land DA experiment. Users who wish to run a more complex experiment may change the values in these files and the files they reference using information in Sections 3.1 & 3.2.
2.1.6. Run an Experiment
The Land DA System uses a script-based workflow that is launched using the do_submit_cycle.sh
script. This script requires a settings_DA_cycle_*
input file that details all the specifics of a given experiment. For example, to run the ERA5 case, users would run:
./do_submit_cycle.sh settings_DA_cycle_era5
Users can replace settings_DA_cycle_era5
with a different settings file to run a different default experiment. Regardless of the file selected, the system will output a message such as Submitted batch job ########
, indicating that the job was successfully submitted. If all goes well, one full cycle will run with data assimilation (DA) and a forecast.
2.1.7. Check Progress
To check on the experiment status, users on a system with a Slurm job scheduler may run:
squeue -u $USER
To view progress, users can open the log*
and err*
files once they have been generated:
tail -f log* err*
Users will need to type Ctrl+C
to exit the files. For examples of what the log and error files should look like in a successful experiment, reference ERA5 Experiment Logs or GSWP3 Experiment Logs below.
Attention
If the log file contains a NetCDF error (e.g., ModuleNotFoundError: No module named 'netCDF4'
), run:
python -m pip install netCDF4
Then, resubmit the job (sbatch submit_cycle.sh
).
Next, check for the background and analysis files in the test directory.
ls -l ../landda_expts/DA_<data_source>_test/mem000/restarts/<vector/tile>``
where:
<data_source>
is eitherera5
orgswp3
, and
<vector/tile>
is eithervector
ortile
depending on whether ERA5 or GSWP3 forcing data was used, respectively.
The experiment should generate several files.
2.1.7.1. ERA5 Experiment Logs
For the ERA5 experiment, the log*
file for a successful experiment will a message like:
Creating: .//ufs_land_restart.2019-12-22_00-00-00.nc
Searching for forcing at time: 2019-12-22 01:00:00
The err*
file for a successful experiment will end with something similar to:
+ THISDATE=2019122200
+ date_count=1
+ '[' 1 -lt 1 ']'
+ '[' 2019122200 -lt 2019122200 ']'
2.1.7.2. GSWP3 Experiment Logs
For the GSWP3 experiment, the log*
file for a successful experiment will end with a list of resource statistics. For example:
Number of times filesystem performed OUTPUT = 250544
Number of Voluntary Context Switches = 3252
Number of InVoluntary Context Switches = 183
*****************END OF RESOURCE STATISTICS*************************
The err*
file for a successful experiment will end with something similar to:
+ echo 'do_landDA: calling apply snow increment'
+ [[ '' =~ hera\.internal ]]
+ /apps/intel-2022.1.2/intel-2022.1.2/mpi/2021.5.1/bin/mpiexec -n 6 /path/to/land-DA_workflow/build/bin/apply_incr.exe /path/to/landda_expts/DA_GSWP3_test/DA/logs//apply_incr.log
+ [[ 0 != 0 ]]
+ '[' YES == YES ']'
+ '[' YES == YES ']'
+ cp /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile1.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile2.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile3.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile4.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile5.nc /path/to/workdir/mem000/jedi/20000103.000000.xainc.sfc_data.tile6.nc /path/to/landda_expts/DA_GSWP3_test/DA/jedi_incr/
+ [[ YES == \N\O ]]