Building the `kaiju` software on `derecho` for MAGE - With TIEGCM (GTR)

Introduction

This page provides instructions for building the kaiju software on the derecho supercomputer. These instructions assume that you have cloned the kaiju repository.

Prepare your software environment

Like most HPC systems, derecho uses the module system to manage the versions of software packages available to the user. When you log in to derecho, the following modules are loaded by default:

module list

Currently Loaded Modules:
  1) ncarenv/23.09  (S)   4) ncarcompilers/1.0.0   7) netcdf/4.9.2
  2) craype/2.7.23        5) cray-mpich/8.1.27
  3) intel/2023.2.1       6) hdf5/1.12.2

  Where:
   S:  Module is Sticky, requires --force to unload or purge

This set of modules cannot be used to build the kaiju code.

Start by purging any currently-loaded modules, then loading the following module set For MAGE runs coupled with TIEGCM (known as “GTR”):

module --force purge
module load ncarenv/23.09
module load cmake/3.26.3
module load craype/2.7.31
module load intel-classic/2023.2.1
module load cray-mpich/8.1.27
module load ncarcompilers/1.0.0
module load mkl/2023.2.0
module load hdf5-mpi/1.12.2
module load netcdf-mpi/4.9.2
module load esmf/8.6.0
module load conda/latest

Important

You must use these exact versions of the modules to ensure the software compiles properly. If you use different versions of any of these modules, a successful build cannot be guaranteed. This module list is current as of 11 April 2025, and is subject to change as the compute environment changes.

Build the `kaiju` software

These instructions show how to build the MPI version of the kaiju software. The GTR version is built in the subdirectory build_gtr under the kaiju source code directory. In practice, you can place the build directory in any convenient location.

# Move to your kaiju clone.
cd /path/to/kaiju

# Create the GTR build directory and enter it.
mkdir build_gtr
cd build_gtr

# Run cmake to create the Makefile, saving output.
# NOTE: The FC definition is *required* for proper cmake operation.
FC=`which ifort` cmake -DENABLE_MPI=ON make -DALLOW_INVALID_COMPILERS=ON .. >& cmake.out

# You can pick one compile target below or compile all of them, if you'd like

# Compile the MAGE model for geospace simulations
make -j4 voltron_mpi.x >& make-voltron.out

# Compile the GAMERA-helio model for inner heliosphere simulations
make -j4 gamhelio_mpi.x >& make-gamhelio.out

# Compile analysis tools
make -j4 calcdb.x chop.x sctrack.x slice.x >& make-analysis.out

When finished, your build directory will contain a bin subdirectory which will contain the compiled kaiju executables.

Build the `tiegcm` software

TIEGCM is a comprehensive, first-principles, three-dimensional, non-linear representation of the coupled thermosphere and ionosphere system that includes a self-consistent solution of the middle and low-latitude dynamo field.

Getting the TIE-GCM source code

The TIE-GCM source code can be obtained by cloning the TIE-GCM repository on GitHub:

git clone https://github.com/NCAR/tiegcm.git

Setting environment variables

Add paths to TIEGCMHOME and TIEGCMDATA. For example:

export TIEGCMHOME=/path/to/your/tiegcm
export TIEGCMDATA=/path/to/your/tiegcm/data

Note

The TIEGCMHOME and TIEGCMDATA environment variables are required for running the GTR model. They should point to the TIEGCM source code directory and the TIEGCM data directory, respectively.

The TIEGCMDATA directory is located in the following locations:

On derecho: /glade/campaign/hao/itmodel/tiegcm3.0/new_data
On pleiades: /nobackup/nrao3/tiegcm/tiegcm3.0/data
The required data files can be downloaded from the NCAR Globus endpoint using the following link: TIEGCM Data Files

Resolution guide for TIEGCM

Two TIEGCM executables are required for running the GTR model:

TIEGCM Standalone
This is the TIEGCM code that runs independently and is used for initialization of the model.

TIEGCM Coupled
This is the TIEGCM code that runs in a coupled mode with the GR model, providing real-time updates to the thermosphere and ionosphere conditions during the simulation.

Depending on the Gamera resolution you will need to compile different TIEGCM resolution executables:

For a D run
- TIEGCM Standalone: horires = 2.5, vertres = 0.25(1/4), mres = 2
- TIEGCM Coupled: horires = 2.5, vertres = 0.25(1/4), mres = 2
For a Q run
- TIEGCM Standalone: horires = 2.5, vertres = 0.25(1/4), mres = 2
- TIEGCM Coupled: horires = 1.25, vertres = 0.125(1/8), mres = 1
For a O run
- TIEGCM Standalone: horires = 1.25, vertres = 0.125(1/8), mres = 1
- TIEGCM Coupled: horires = 0.625, vertres = 0.0625(1/16), mres = 0.5

The TIEGCM code is built using the tiegcmrun script, which is provided in the tiegcm code repository. The script is provided in the tiegcm/tiegcmrun directory. More information on tiegcmrun.py can be found in the TIEGCM Quick Start Guide.

Important

Make sure to load the modules lised in the kaiju build instructions before running the tiegcmrun script.

Build guide for TIEGCM code for a `Q` run:

We will use tiegcmrun script to build the code which requires the minimum amount of input from the user. At each prompt, you can either type in a value, or hit the Return key to accept the default value (shown in square brackets at the end of the prompt).

First we will create a directory for the “Q” TIEGCM build in the TIEGCMHOME directory.

cd $TIEGCMHOME
mkdir tiegcm_build_Q
cd tiegcm_build_Q

Next, we will build the standalone TIEGCM executable by running the tiegcmrun.py script with the -oc.

Note

The -oc option stands for “only compile”, which means that the script will only compile the code and not run it. Since the Gamera resolution is Q, we will set the following:

horizontal resolution for the standalone TIEGCM to 2.5 degrees

vertical resolution to 0.25(1/4) scale height

magnetic grid resolution to 2 degrees

$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description


Run Options:
User Mode = BASIC
Compile = True
Execute = False
Coupling = False


Name of HPC system (derecho|pleiades|linux) [derecho]:
Standalone Executable [/glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.exe]:
Horizontal Resolution (Deg) (5.0|2.5|1.25|0.625) [2.5]:
Vertical Resolution (Scale Height) (1/2|1/4|1/8|1/16) [1/4]:
Magnetic grid resolution (Degree) (2|1|0.5) [2]:

After these inputs, the script will compile the TIEGCM code and create the standalone executable and should output something like this:

..
..
gmake[1]: Leaving directory '/glade/derecho/scratch/nikhilr/tiegcm_build/exec'
Executable copied from /glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.exe to /glade/derecho/scratch/nikhilr/tiegcm_build/stdout

Next, we will build the coupled TIEGCM executable by running the tiegcmrun.py script with the -oc and -co options.

Note

The -co option stands for “coupled”, which means that the script will compile the code for the coupled TIEGCM executable. Since the Gamera resolution is Q, we will set the following:

horizontal resolution for the coupled TIEGCM to 1.25 degrees

vertical resolution to 0.125(1/8) scale height

magnetic grid resolution to 1 degrees

$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description

Run Options:
User Mode = BASIC
Compile = True
Execute = False
Coupling = True

Name of HPC system (derecho|pleiades|linux) [derecho]:
Coupled Executable [/glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.x]:
Horizontal Resolution (Deg) (5.0|2.5|1.25|0.625) [2.5]: 1.25
Vertical Resolution (Scale Height) (1/2|1/4|1/8|1/16) [1/8]:
Magnetic grid resolution (Degree) (2|1|0.5) [1]:

After these inputs, the script will compile the TIEGCM code and create the coupled executable and should output something like this:

..
..
gmake[1]: Leaving directory '/glade/derecho/scratch/nikhilr/tiegcm_build/exec'
Executable copied from /glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.x to /glade/derecho/scratch/nikhilr/tiegcm_build/stdout

You should now see the following files in your run directory:

ls
exec  hist  stdout

The executables are located in the stdout directory, and the stdout files

ls stdout
defs.h  tiegcm.exe  tiegcm.x

Build guide for TIEGCM code for a `D` run on `derecho`:

1. First we will create a directory for the “D” TIEGCM build in the TIEGCMHOME directory. .. code-block:: bash

cd $TIEGCMHOME mkdir tiegcm_build_D cd tiegcm_build_D

Next, we will build the standalone TIEGCM executable by running the tiegcmrun.py script with the -oc.

Note

Since the Gamera resolution is D, we will set the following:

horizontal resolution for the standalone TIEGCM to 2.5 degrees

vertical resolution to 0.25(1/4) scale height

magnetic grid resolution to 2 degrees

$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc

Next, we will build the coupled TIEGCM executable by running the tiegcmrun.py script with the -oc and -co options.

Note

Since the Gamera resolution is D, we will set the following:

horizontal resolution for the coupled TIEGCM to 2.5 degrees

vertical resolution to 0.25(1/4) scale height

magnetic grid resolution to 2 degrees

$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co

The executables are located in the stdout directory, and the stdout files

ls stdout
defs.h  tiegcm.exe  tiegcm.x

Build guide for TIEGCM code for a `O` run on `derecho`:

1. First we will create a directory for the “D” TIEGCM build in the TIEGCMHOME directory. .. code-block:: bash

cd $TIEGCMHOME mkdir tiegcm_build_O cd tiegcm_build_O

Next, we will build the standalone TIEGCM executable by running the tiegcmrun.py script with the -oc.

Note

Since the Gamera resolution is O, we will set the following:

horizontal resolution for the standalone TIEGCM to 1.25 degrees

vertical resolution to 0.125(1/8) scale height

magnetic grid resolution to 1 degrees

$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc

Next, we will build the coupled TIEGCM executable by running the tiegcmrun.py script with the -oc and -co options.

Note

Since the Gamera resolution is O, we will set the following:

horizontal resolution for the coupled TIEGCM to 0.625 degrees

vertical resolution to 0.0625(1/16) scale height

magnetic grid resolution to 0.5 degrees

$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co

The executables are located in the stdout directory, and the stdout files

ls stdout
defs.h  tiegcm.exe  tiegcm.x

Note

Documentation on the analysis tools is found here.

Building the kaiju software on derecho for MAGE - With TIEGCM (GTR)

Introduction

Prepare your software environment

Build the kaiju software

Build the tiegcm software

Getting the TIE-GCM source code

Setting environment variables

Resolution guide for TIEGCM

Build guide for TIEGCM code for a Q run:

Build guide for TIEGCM code for a D run on derecho:

Build guide for TIEGCM code for a O run on derecho:

Building the `kaiju` software on `derecho` for MAGE - With TIEGCM (GTR)

Build the `kaiju` software

Build the `tiegcm` software

Build guide for TIEGCM code for a `Q` run:

Build guide for TIEGCM code for a `D` run on `derecho`:

Build guide for TIEGCM code for a `O` run on `derecho`: