Compiling the kaiju software on CentOS-Stream 9

Introduction

These instructions will walk you through the process of building and installing the kaiju software on a CentOS-Stream 9 system.

These instructions assume that the user is using the bash shell, and that no modifications have been made to the user “dotfiles” ($HOME/.bashrc, $HOME/.bash_profile). If you have customized either of these files for your account, please carefully save and inspect the output from each command in the build process to ensure that no unexpected problems have crept in. To facilitate this practice, all of the commands shown below will illustrate how to save command output, and how to measure how long each step takes. The latter is a useful bit of information which can help identify build problems early in the process, avoiding much wasted time and effort later.

In the instructions below, all code will be built and installed in a CentOS-Stream 9-specific subdirectory of the user home directory, i.e. $HOME/centos-stream-9. This particular organization is not required - it is intended as an example of one possible way to segregate software that has been built for multiple systems.

Building the serial version of the kaiju software on CentOS-Stream 9

Step 1: Configure build tools

Begin by configuring tools and libraries needed for the build:

# Add the Intel compilers to PATH.
source /opt/intel/oneapi/setvars.sh

# Set variables for HDF5.
export HOST_SYSTEM=centos-stream-9
export HDF5_DIR=$HOME/$HOST_SYSTEM/local/hdf5/1.14.1-2
export PATH=$HDF5_DIR/bin:$PATH
export HDF5_INCLUDE_DIRS=$HDF5_DIR/include
export HDF5_LIBRARIES=$HDF5_DIR/lib
export CPATH=$HDF5_INCLUDE_DIRS
export INCLUDE="-I$HDF5_INCLUDE_DIRS"

# Configure CDF.
source $HOME/$HOST_SYSTEM/local/cdf/3.9.0/bin/definitions.B

Step 2: Create the build directory

Create a system-specific build directory.

cd $HOME
mkdir HOST_SYSTEM
cd $HOST_SYSTEM

Then make an additional subdirectory level for the branch of the code you are building (the development branch is used as an example). This arrangement is useful when you need to maintain simultaneous builds of different branches.

code-block:: shell

export KAIJU_BRANCH_NAME=development mkdir $KAIJU_BRANCH_NAME cd $KAIJU_BRANCH_NAME

Step 3: Clone the kaiju repository from BitBucket

NOTE: This step assumes you have been granted access to the kaiju repository on BitBucket, and that you have configured an SSH key pair for use with BitBucket. If you need help with these tasks, please contact a CGS team member for assistance.

Clone the kaiju repository (or “repo”) from BitBucket:

git clone git@bitbucket.org:aplkaiju/kaiju.git

This process should take a minute or so. When complete, verify that the kaiju code exists in your directory (the actual directory contents may differ slightly from what is shown below):

ls kaiju
analysis  cmake  CMakeLists.txt  examples  external  gitHookScripts  kaiju.sublime-project  kaipy  places  pytests  quickstart  README.md  scripts  setup.py  src  testingScripts  tests  xml

Now move down into the cloned repo, and switch to the branch of the code you wish to use. By default, the cloned repository provides the master branch, but we want the development branch:

cd kaiju
git switch $KAIJU_BRANCH_NAME

Step 4: Run cmake to create the Makefile needed to build the software

Since the kaiju code can be built in serial and MPI forms, we first make a directory in which to build the serial version of the code (use whatever name you prefer, but build_serial is simple and unambiguous):

export KAIJU_BUILD_NAME=build_serial
KAIJU_BUILD_PATH=$KAIJU_HOME/$KAIJU_BUILD_NAME
mkdir -p $KAIJU_BUILD_PATH
cd $KAIJU_BUILD_PATH

Now run the cmake command. Save the cmake output, and use timestamps for each step. The options shown below direct the build process to use a recent version of the Intel Fortran compiler:

date; time FC=`which ifort` cmake -DALLOW_INVALID_COMPILERS=ON >& cmake.out; date

This command usually takes 2-3 seconds, depending on system activity. Examine the output file cmake.out for problems. It should look something like this:

-- The Fortran compiler identification is Intel 2021.10.0.20230609
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort - skipped
-- Checking whether /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort supports Fortran 90
-- Checking whether /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort supports Fortran 90 - yes
-- Found HDF5: /home/ewinter/centos-stream-9/local/hdf5/1.14.1-2/lib/libhdf5_fortran.so;/home/ewinter/centos-stream-9/local/hdf5/1.14.1-2/lib/libhdf5.so;/usr/lib64/libz.so;/usr/lib64/libdl.a;/usr/lib64/libm.so (found version "1.14.1-2") found components: Fortran
-- Found OpenMP_Fortran: -qopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0") found components: Fortran
CMake Warning at cmake/compilers.cmake:61 (message):
  Setting default optimization to O2 to avoid certain Intel compiler bugs
Call Stack (most recent call first):
  CMakeLists.txt:99 (include)


-------------------------
Configuration summary ...
System: localhost.localdomain
OS: Linux
Processor: x86_64
Compiler: Intel / 2021.10.0.20230609
          /opt/intel/oneapi/compiler/2023.2.0/linux/bin/intel64/ifort
HDF5 Wrapper: /home/ewinter/centos-stream-9/local/hdf5/1.14.1-2/bin/h5fc
Version: 3f4e147c / development
Build Type: Release
Base Flags:  -fPIC -free -implicitnone -qopenmp
Build Flags: -O2 -align array64byte -align rec32byte -no-prec-div -fast-transcendentals -ipo
-------------------------

Adding CHIMP module ...
    EB IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/chimp/ebICs/ebICstd.F90
    TP IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/chimp/tpICs/tpICstd.F90
    Adding executable project.x
    Adding executable psd.x
    Adding executable push.x
    Adding executable slice.x
    Adding executable chop.x
    Adding executable trace.x
    Adding executable sctrack.x
    Adding executable calcdb.x
    Adding executable wpicheck.x
Adding Gamera module ...
    Bricksize is 16
    IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/gamera/ICs/null.F90
    Adding executable gamera.x
Adding Gamera Helio module ...
    IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/gamera/ICs/helio/wsa.F90
    Adding executable gamhelio.x
Adding ReMIX module ...
    Adding executable remix.x
Adding RCM module ...
    RCM Grid is of size 180 x 361 x 160
    Adding executable rcm.x
Adding Voltron module ...
    IC file is /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/src/voltron/ICs/earthcmi.F90
    Adding executable voltron.x
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ewinter/centos-stream-9/cgs/kaiju/development/kaiju/build_serial

Step 5: Compile the kaiju software

Now use make to build the kaiju software, time-stamping and saving the output:

date; time make >& make.out; date

This command should complete in about 8-9 minutes on a CentOS-Stream 9 system running on an i9 processor. When the command is finished, check the output file make.out. The file is long, but the last few lines should look something like this:

[ 99%] Linking Fortran executable bin/sctrack.x
[ 99%] Built target sctrack.x
[ 99%] Built target chimp
[ 99%] Built target voltron
Scanning dependencies of target remix.x
[100%] Building Fortran object CMakeFiles/remix.x.dir/src/drivers/remix.F90.o
[100%] Linking Fortran executable bin/remix.x
[100%] Built target remix.x
[100%] Built target remix
[100%] Built target gamera

To verify that all of the kaiju programs have been built, examine the bin subdirectory of your build_serial directory (this list will evolve as more programs are added):

ls bin
calcdb.x  gamhelio.x  psd.x   remix2rcm.x    sctrack.x  voltron.x
chop.x    kaitoy.x    push.x  remix2remix.x  slice.x    wpicheck.x
gamera.x  project.x   rcm.x   remix.x        trace.x

Using the kaiju software

Once built, you must run the setup script before using the kaiju software:

source $KAIJU_HOME/scripts/setupEnvironment.sh

This script will set environment variables needed by the kaiju software, including the KAIJUHOME environment variable (not the KAIJU_HOME environment variable). However, the path to the compiled programs is not added - you will need to specify the complete path when using compiled programs. For example, to run the serial version of gamera.x:

$KAIJUHOME/build_serial/bin/gamera.x