Building the ``kaiju`` software on ``derecho`` for MAGE - With TIEGCM (GTR)
==============================================================================================
Introduction
------------
This page provides instructions for building the ``kaiju`` software on the
``derecho`` supercomputer. These instructions assume that you have cloned the
``kaiju`` repository.
Prepare your software environment
---------------------------------
Like most HPC systems, ``derecho`` uses the ``module`` system to manage the
versions of software packages available to the user. When you log in to
``derecho``, the following modules are loaded by default:
.. code-block:: bash
module list
Currently Loaded Modules:
1) ncarenv/23.09 (S) 4) ncarcompilers/1.0.0 7) netcdf/4.9.2
2) craype/2.7.23 5) cray-mpich/8.1.27
3) intel/2023.2.1 6) hdf5/1.12.2
Where:
S: Module is Sticky, requires --force to unload or purge
This set of modules **cannot** be used to build the ``kaiju`` code.
Start by purging any currently-loaded modules, then loading the following
module set For MAGE runs coupled with TIEGCM (known as "GTR"):
.. code-block:: bash
module --force purge
module load ncarenv/23.09
module load cmake/3.26.3
module load craype/2.7.31
module load intel-classic/2023.2.1
module load cray-mpich/8.1.27
module load ncarcompilers/1.0.0
module load mkl/2023.2.0
module load hdf5-mpi/1.12.2
module load netcdf-mpi/4.9.2
module load esmf/8.6.0
module load conda/latest
.. important::
You must use these exact versions of the modules to ensure the software
compiles properly. If you use different versions of any of these modules,
a successful build cannot be guaranteed. This module list is current as of
**11 April 2025**, and is subject to change as the compute environment
changes.
Build the ``kaiju`` software
----------------------------
These instructions show how to build the MPI version of the ``kaiju``
software. The GTR version is built in the subdirectory ``build_gtr``
under the ``kaiju`` source code directory. In practice, you can place the
build directory in any convenient location.
.. code-block:: bash
# Move to your kaiju clone.
cd /path/to/kaiju
# Create the GTR build directory and enter it.
mkdir build_gtr
cd build_gtr
# Run cmake to create the Makefile, saving output.
# NOTE: The FC definition is *required* for proper cmake operation.
FC=`which ifort` cmake -DENABLE_MPI=ON make -DALLOW_INVALID_COMPILERS=ON .. >& cmake.out
# You can pick one compile target below or compile all of them, if you'd like
# Compile the MAGE model for geospace simulations
make -j4 voltron_mpi.x >& make-voltron.out
# Compile the GAMERA-helio model for inner heliosphere simulations
make -j4 gamhelio_mpi.x >& make-gamhelio.out
# Compile analysis tools
make -j4 calcdb.x chop.x sctrack.x slice.x >& make-analysis.out
When finished, your build directory will contain a ``bin``
subdirectory which will contain the compiled ``kaiju`` executables.
Build the ``tiegcm`` software
-----------------------------
`TIEGCM `_ is a comprehensive, first-principles, three-dimensional,
non-linear representation of the coupled thermosphere and ionosphere system that includes a self-consistent solution
of the middle and low-latitude dynamo field.
Getting the TIE-GCM source code
************************************************
The ``TIE-GCM`` source code can be obtained by cloning the ``TIE-GCM`` repository
on GitHub:
.. code-block:: bash
git clone https://github.com/NCAR/tiegcm.git
Setting environment variables
************************************************
Add paths to ``TIEGCMHOME`` and ``TIEGCMDATA``.
For example:
.. code-block:: bash
export TIEGCMHOME=/path/to/your/tiegcm
export TIEGCMDATA=/path/to/your/tiegcm/data
.. note::
The ``TIEGCMHOME`` and ``TIEGCMDATA`` environment variables are required
for running the GTR model. They should point to the TIEGCM source code
directory and the TIEGCM data directory, respectively.
The TIEGCMDATA directory is located in the following locations:
- On ``derecho``: ``/glade/campaign/hao/itmodel/tiegcm3.0/new_data``
- On ``pleiades``: ``/nobackup/nrao3/tiegcm/tiegcm3.0/data``
- The required data files can be downloaded from the NCAR Globus endpoint using the following link: `TIEGCM Data Files `_
Resolution guide for TIEGCM
************************************************
Two TIEGCM executables are required for running the GTR model:
- TIEGCM Standalone
This is the TIEGCM code that runs independently and is used for initialization of the model.
- TIEGCM Coupled
This is the TIEGCM code that runs in a coupled mode with the GR model, providing
real-time updates to the thermosphere and ionosphere conditions during the simulation.
Depending on the Gamera resolution you will need to compile different TIEGCM resolution executables:
- For a ``D`` run
- TIEGCM Standalone: horires = 2.5, vertres = 0.25(1/4), mres = 2
- TIEGCM Coupled: horires = 2.5, vertres = 0.25(1/4), mres = 2
- For a ``Q`` run
- TIEGCM Standalone: horires = 2.5, vertres = 0.25(1/4), mres = 2
- TIEGCM Coupled: horires = 1.25, vertres = 0.125(1/8), mres = 1
- For a ``O`` run
- TIEGCM Standalone: horires = 1.25, vertres = 0.125(1/8), mres = 1
- TIEGCM Coupled: horires = 0.625, vertres = 0.0625(1/16), mres = 0.5
The TIEGCM code is built using the ``tiegcmrun`` script, which is provided in
the ``tiegcm`` code repository. The script is provided in the
``tiegcm/tiegcmrun`` directory. More information on ``tiegcmrun.py`` can be found
in the `TIEGCM Quick Start Guide `_.
.. important::
Make sure to load the modules lised in the ``kaiju`` build instructions
before running the ``tiegcmrun`` script.
Build guide for TIEGCM code for a ``Q`` run:
#########################################################################################
We will use ``tiegcmrun`` script to build the code which requires the minimum amount
of input from the user. At each prompt, you can either type in a value, or hit
the :kbd:`Return` key to accept the default value (shown in square brackets at
the end of the prompt).
1. First we will create a directory for the "Q" TIEGCM build in the TIEGCMHOME directory.
.. code-block:: bash
cd $TIEGCMHOME
mkdir tiegcm_build_Q
cd tiegcm_build_Q
2. Next, we will build the standalone TIEGCM executable by running the ``tiegcmrun.py`` script with the ``-oc``.
.. note::
The ``-oc`` option stands for "only compile", which means that the script will only compile the code and not run it.
Since the Gamera resolution is ``Q``, we will set the following:
- horizontal resolution for the standalone TIEGCM to 2.5 degrees
- vertical resolution to 0.25(1/4) scale height
- magnetic grid resolution to 2 degrees
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description
Run Options:
User Mode = BASIC
Compile = True
Execute = False
Coupling = False
Name of HPC system (derecho|pleiades|linux) [derecho]:
Standalone Executable [/glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.exe]:
Horizontal Resolution (Deg) (5.0|2.5|1.25|0.625) [2.5]:
Vertical Resolution (Scale Height) (1/2|1/4|1/8|1/16) [1/4]:
Magnetic grid resolution (Degree) (2|1|0.5) [2]:
After these inputs, the script will compile the TIEGCM code and create the standalone executable and should output something like this:
.. code-block:: bash
..
..
gmake[1]: Leaving directory '/glade/derecho/scratch/nikhilr/tiegcm_build/exec'
Executable copied from /glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.exe to /glade/derecho/scratch/nikhilr/tiegcm_build/stdout
3. Next, we will build the coupled TIEGCM executable by running the ``tiegcmrun.py`` script with the ``-oc`` and ``-co`` options.
.. note::
The ``-co`` option stands for "coupled", which means that the script will compile the code for the coupled TIEGCM executable.
Since the Gamera resolution is ``Q``, we will set the following:
- horizontal resolution for the coupled TIEGCM to 1.25 degrees
- vertical resolution to 0.125(1/8) scale height
- magnetic grid resolution to 1 degrees
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co
Instructions:
-> Default Selected input parameter is given in GREEN
-> Warnings and Information are given in YELLOW
-> Errors are given in RED
-> Valid values (if any) are given in brackets eg. (value1 | value2 | value3)
-> Enter '?' for any input parameter to get a detailed description
Run Options:
User Mode = BASIC
Compile = True
Execute = False
Coupling = True
Name of HPC system (derecho|pleiades|linux) [derecho]:
Coupled Executable [/glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.x]:
Horizontal Resolution (Deg) (5.0|2.5|1.25|0.625) [2.5]: 1.25
Vertical Resolution (Scale Height) (1/2|1/4|1/8|1/16) [1/8]:
Magnetic grid resolution (Degree) (2|1|0.5) [1]:
After these inputs, the script will compile the TIEGCM code and create the coupled executable and should output something like this:
.. code-block:: bash
..
..
gmake[1]: Leaving directory '/glade/derecho/scratch/nikhilr/tiegcm_build/exec'
Executable copied from /glade/derecho/scratch/nikhilr/tiegcm_build/exec/tiegcm.x to /glade/derecho/scratch/nikhilr/tiegcm_build/stdout
4. You should now see the following files in your run directory:
.. code-block:: bash
ls
exec hist stdout
The executables are located in the ``stdout`` directory, and the stdout files
.. code-block:: bash
ls stdout
defs.h tiegcm.exe tiegcm.x
Build guide for TIEGCM code for a ``D`` run on ``derecho``:
#########################################################################################
1. First we will create a directory for the "D" TIEGCM build in the TIEGCMHOME directory.
.. code-block:: bash
cd $TIEGCMHOME
mkdir tiegcm_build_D
cd tiegcm_build_D
2. Next, we will build the standalone TIEGCM executable by running the ``tiegcmrun.py`` script with the ``-oc``.
.. note::
Since the Gamera resolution is ``D``, we will set the following:
- horizontal resolution for the standalone TIEGCM to 2.5 degrees
- vertical resolution to 0.25(1/4) scale height
- magnetic grid resolution to 2 degrees
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc
3. Next, we will build the coupled TIEGCM executable by running the ``tiegcmrun.py`` script with the ``-oc`` and ``-co`` options.
.. note::
Since the Gamera resolution is ``D``, we will set the following:
- horizontal resolution for the coupled TIEGCM to 2.5 degrees
- vertical resolution to 0.25(1/4) scale height
- magnetic grid resolution to 2 degrees
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co
4. The executables are located in the ``stdout`` directory, and the stdout files
.. code-block:: bash
ls stdout
defs.h tiegcm.exe tiegcm.x
Build guide for TIEGCM code for a ``O`` run on ``derecho``:
#########################################################################################
1. First we will create a directory for the "D" TIEGCM build in the TIEGCMHOME directory.
.. code-block:: bash
cd $TIEGCMHOME
mkdir tiegcm_build_O
cd tiegcm_build_O
2. Next, we will build the standalone TIEGCM executable by running the ``tiegcmrun.py`` script with the ``-oc``.
.. note::
Since the Gamera resolution is ``O``, we will set the following:
- horizontal resolution for the standalone TIEGCM to 1.25 degrees
- vertical resolution to 0.125(1/8) scale height
- magnetic grid resolution to 1 degrees
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc
3. Next, we will build the coupled TIEGCM executable by running the ``tiegcmrun.py`` script with the ``-oc`` and ``-co`` options.
.. note::
Since the Gamera resolution is ``O``, we will set the following:
- horizontal resolution for the coupled TIEGCM to 0.625 degrees
- vertical resolution to 0.0625(1/16) scale height
- magnetic grid resolution to 0.5 degrees
.. code-block:: bash
$TIEGCMHOME/tiegcmrun/tiegcmrun.py -oc -co
4. The executables are located in the ``stdout`` directory, and the stdout files
.. code-block:: bash
ls stdout
defs.h tiegcm.exe tiegcm.x
.. note:: Documentation on the analysis tools is found
:doc:`here `.