calcdb.x - Computing ground delta-B values from kaiju results

Introduction

The tool calcdb.x is a compiled Fortran program which uses the output from a MAGE simulation to compute the values of the ground magnetic field disturbance.

Before you start

In order to use calcdb.x, you need the following:

  • A set of output files from a kaiju magnetosphere run.

  • An XML file describing the magnetosphere results and the settings for the ground delta B calculation.

  • (optional) A PBS job file to run calcdb.x.

kaiju magnetosphere result files

Any set of kaiju magnetosphere results can be used. This documentation assumes you used the MPI version the the kaiju software, e.g. gamera_mpi.x. This should generate a number of HDF5 files which contain the model results from each MPI rank.

Assume all result files are in the current directory.

The XML file for calcdb.x

The XML file read by calcdb.x should look something like this (file calcdb.xml):

<?xml version="1.0"?>
<Kaiju>
    <Chimp>
        <sim runid="geospace"/>
        <time T0="0.0" dt="60.0" tFin="14400.0"/>
        <fields ebfile="geospace" grType="LFM" doJ="T" isMPI="true"/>
        <parallel Ri="4" Rj="4" Rk="1"/>
        <grid Nlat="360" Nlon="720" Nz="1"/>
        <parintime NumB="4"/>
    </Chimp>
</Kaiju>

The elements and attributes of the XML file are described below.

Note

Standard XML is case-sensitive. However, the XML parser used by the Fortran code in kaiju is case-insensitive. Therefore, all of the XML tags and attributes described below can be written with any combintion of upper and lower case.

  • <?xml version="1.0"?> (required): Specifies XML version.

  • <Kaiju> (required): Outer element for all kaiju elements.

  • <Chimp> (required): Inner element for CHIMP-specific elements.

  • <sim> (optional): Specify identifying information for the kaiju results to use,

    • runid (optional, default "Sim"): String specifying the runid for the kaiju results to use in the ground delta B calculation.

  • <time> (optional): Specify time range and interval for magnetic field calculation.

    • T0 (optional, default "0.0"): Start time (simulated seconds) for ground magnetic field calculation, relative to start of simulation results used as input.

    • dt (optional, default "1.0"): Time interval and output cadence (simulated seconds) for ground magnetic field calculation.

    • tFin (optional, default "60.0"): Stop time (simulated seconds) for ground magnetic field calculation, relative to start of simulation results used as input.

  • <fields> (required): Describes the MAGE model results to use.

    • ebfile (optional, default "ebdata"): Root name for HDF5 files containing the results produced by a MAGE model run. This is usually the same as the runid.

    • grType (optional, default "EGG"): String specifying grid type used by the MAGE output files. Valid values are "EGG", "LFM", and "SPH". If the string is not one of the supported grid types, the default value ("EGG") is used, and a warning message is printed. Note that for a magnetosphere simulation, the grType is always "LFM".

    • doJ (required, must be "T"): If "T", compute currents from the kaiju model results.

    • isMPI (optional): If "true", the MAGE results are from an MPI run.

  • <parallel> (required): Describes the MPI decomposition of the MAGE model run.

    • Ri (optional, default "1"): Number of ranks used in MPI decomposition of i dimension.

    • Rj (optional, default "1"): Number of ranks used in MPI decomposition of j dimension.

    • Rk (optional, default "1"): Number of ranks used in MPI decomposition of k dimension.

  • <grid> (optional): Options to specify the grid on the ground used in calcdb.x.

    • Nlat (optional, default "45"): Number of latitude cells.

    • Nlon (optional, default "90"): Number of longitude cells.

    • Nz (optional, default "2"): Number of altitude cells.

Running calcdb.x

We can now run calcdb.x with the following command:

calcdb.x calcdb.xml

For large simulations, this process can take a long time. A more efficient approach is to use PBS to run multiple calcdb.x jobs in parallel, and combine the results. To do this, add the following element to the XML file. in the <Chimp> element:

<parintime NumB="4"/>

where:

  • <parintime> (optional): Options to run a PBS job array of calcdb.x to increase calculation speed.

    • NumB (optional, default "0"): Number of jobs into which the calculation will be split for parallel computation.

We also need a PBS script, as shown below (the example runs on derecho):

#PBS -a YOUR_PBS_ACCOUNT
#PBS -q main
#PBS -l walltime=01:00:00
#PBS -l select=1:ncpus=128
#PBS -l job_priority=economy

module purge
# Load your modules here.
module list

export KAIJU_INSTALL_DIR=/path/to/your/kaiju/code
source $KAIJU_INSTALL_DIR/scripts/setupEnvironment.sh
export KAIJU_BUILD_DIR=/path/to/your/kaiju/build
cp $KAIJU_BUILD_DIR/bin/calcdb.x ./calcdb.x

export JNUM=${PBS_ARRAY_INDEX:-0}
./calcdb.x calcdb.xml $JNUM

Now submit the job to the PBS system:

$ qsub -J 1-4 calcdb.pbs

Note

The second digit in -J 1-4 must match the value of NumB in the XML file.

When the calculation has finished, your directory will contain a set of files named like this:

ls -1 *.deltab.h5
geospace.0001.deltab.h5
geospace.0002.deltab.h5
geospace.0003.deltab.h5
geospace.0004.deltab.h5

Now combine the results into a single file with the pitmerge.py command (pitmerge.py is provided in the kaipy python package):

pitmerge.py -runid geospace

where runid is the runid for the kaiju run being used for the calculation. This command will create the file geospace.deltab.h5, which contains the ground delta B values for the simulation.