GFDL - Geophysical Fluid Dynamics Laboratory

FRE Installation Guide

Contents

Introduction

FRE is multi-language, multi-site workflow manager for the Flexible Modeling System (FMS). It consists of a collection of utilities for compiling and running climate simulations.

When porting FRE, it is necessary to consider: the system architecture, the operating system version, and available system resources (schedulers, filesystems, etc). FRE is designed for use on Linux based systems.

Quickstart

If your goal is to quickly setup and run FRE, read this section. If you’d like to learn more about FRE’s internals and conventions, please start reading the next section.

Installation

Acquire the Release

FRE is released as: fre-commands-bronx-public.tar.gz and fre-combiners-bronx-public.tar.gz

Install the prerequisites

Most prerequisites require the developer version of the package.

  • Fortran Compiler
  • C Compiler
  • Environment Modules (with Tcl)
  • Perl CPAN
  • Perl package “Date::Manip”
  • expat library
  • zlib library
  • Mpich2 library

Unpack the tarball

 tar xzf fre-bronx-public.tar.gz

Download third party tools and move them to the zip folder

For convenience, we’ve provided a set of third party tools and installation scripts for prerequisites less likely to be available already on your systems. We also include netcdf, as we’ve found some netcdf installations on sites we’ve tested don’t support FMS and FRE. See Prerequisite Software for the list of tools. To have the FRE installer install these tools, please download fre-bronx-public-zip.tar.gz (26.5 MB), extract the tar file, and place the third party files inside the main FRE release package at fre/bronx-public/zip, where the FRE installer will look for them.

mv fre-bronx-public-zip/* fre/bronx-public/zip/

Create a new directory where you’d like to install FRE

 mkdir ~/FRE


Copy the modulefiles directory from the unpacked tarball into the new directory

 cp -r modulefiles ~/FRE

Your shell will need to be able to locate the modulefiles. Here’s an example of what the module command you would use to tell your environment where to look for your modulefiles:

 module use ~/FRE/modulefiles

Edit the fre-system/init modulefile

This file configures FRE related directories and sets the required ‘site’ variable which is used throughout FRE.

 vi ~/FRE/modulefiles/fre-system/init

Four variables are required.

  • “site” is a site label which you will use during the installation.
  • “zipDir” is the location of the tar files to be installed with your FRE installation.
  • “sysDir” is the location where the compiled FRE tools will be installed
  • “tmpDir” is the location for temporary files required by FRE (which can potentially be quite large, so choose an appropriate directory). Please note that tmpDir is not where temporary files are stored during the compilation of the various utilities. Those files are instead stored in /tmp.

Here is a very simple example:

   set site example
   set zipDir ~/FRE/soft
   set sysDir ~/FRE/opt
   set tmpDir ~/FRE/tmp


Depending on the compiler you are using, edit that file in fre-system/<compiler>

This file sets environment variables related to the compiler you’d like to use in FRE. Here we have chosen the gnu compilers as an example:

 vi ~/FRE/modulefiles/fre-system/gnu-4

If needed, perform any ‘module loads’ or ‘module unloads’ for the compiler here.

Edit fre-system/netcdf-4-static

This file is an intermediate layer between the various netcdf versions that may be installed on the system. It loads netcdf related environment variables.

vi ~/FRE/modulefiles/fre-system/netcdf-4-static

Set compiler options for your netcdf installation and module loads/unloads for netcdf. You can use a system provided netcdf version or rely on the version provided by the FRE installation. We advise using the FRE provided version. A simple example that is provided as a template for the FRE-provided version.

   module load netcdf/4.1.1-${env(FRE_SYSTEM_COMPILER)}
   set hdf5Home ${env(HDF5_HOME)}
   set netcdfHome ${env(NETCDF_HOME)}
   set lddFlags ""
   set clibsStatic "-lnetcdf -lhdf5_hl -lhdf5"
   set flibsStatic "-lnetcdff -lnetcdf -lhdf5_hl -lhdf5"
   set libsDynamic "-lz"
   set hardwareDir ${hardware}

Load the setup modules

Now that all of the initial configuration has been completed, we’ll load these modules into our environment. Note that we do not need the netcdf module at this time.

module load fre-system/init
module load fre-system/gnu-4 # along with any needed system modules for your compiler

Run the install script

From the untarred FRE directory, run the install script:

./fre-install
  • If any errors are detected, correct them and re-run the install script. It will prompt you for some information; the default is generally acceptable.

A list of the software that will be installed can be found in Prerequisite Software.

Configuration

Now that FRE is installed, we need to prepare it for running a model.

Load FRE

Load FRE into the environment.

module load fre

Create your FRE site

You need to configure a ‘site’ before using FRE. When you configured the fre-system/init file, you set the site name variable (we used ‘sample-site’ in the example). You must usethis same name in the following steps.

cd ~/FRE/opt/fre-commands/bronx-public/site
cp -r example sample-site
cd sample-site

For a simple site, some changes to fre.properties may be all that is required. However, the following files within the site directory may need to be modified.

fre.properties

This file contains name-value pairs that FRE uses internally. It defines directory locations and special options within FRE. See comments in the example site that start with “#Customize” — those are the most important lines to customize for your site.

hsmget.inc and hsmput.inc

You likely do not need to edit these files, but internally FRE refers to two tools called hsmget/hsmput (hierarchical storage manager get and put). The settings in these files determine which copy and file manipulation commands these tools should use. The defaults are all standard unix utilities, so no changes are likely to be needed.

runscript.template.main

This file contains settings for gridspec and initial condition files. It likely does not require modification.

<compiler>.mk

There are a number of Makefiles for holding compiler options. If you do not have a particular compiler, or do not plan to use it for compilation, you do not need to edit that compilers Makefile.

If you run into compilation problems, you may need to modify your library locations using the LIBS or LDFLAGS variables. The default gnu.mk file attempts to use nc-config and pkg-config to determine library locations.

Load NetCDF

NetCDF is required for most FRE operations, so we can load it into the environment at this time. The command below loads the FRE supplied netcdf for the GNU compiler:

module load netcdf/4.1.1-gnu-4

Modify your XML file

Sample experiment XMLs are provided in your FRE installation directory. In our example, they are stored in:

cd ~/FRE/xml

Let’s use the atm.xml as an example. You’ll need to add your site to the <setup> section of the XML file. A very simple platform might look like:

<platform name="sample-site.default">
     <directory stem="$(FRE_STEM)"/>
     <csh><![CDATA[                                                                                                                                    
       source $MODULESHOME/init/csh                                                                                                                    
       module use /home/$USER/FRE/modulefiles                                                                                                      
       module purge                                                                                                                                    
       module load fre                                                                                                                                 
       module load netcdf/4.1.1-gnu-4                                                                                                                  
      ]]>
     </csh>
     <property name="FMS_ARCHIVE_ROOT" value="/home/$USER/FRE/input"/>
</platform>

The above platform is simply instructing the Environment Modules system on where to look for our new FRE modulefiles, and then loading fre and netcdf. The <property> tag is a name-value pair that FRE uses for substitutions. Anywhere that $(FMS_ARCHIVE_ROOT) is defined in our XML, FRE will replace it with “/home/$USER/FRE/input”. The ~/FRE/input directory is a new directory that will hold the experiment’s input data files.

You may need to add commands to module load compilers or MPI/mpich to the platform csh.

Running frelist

We can run the frelist command to obtain a list of which experiments exist in the XML.

frelist -x atm.xml

Running fremake

We can now compile one of our experiment using fremake. Let’s choose the HSFvd experiment that we noticed after running frelist on atm.xml:

fremake -x ~/FRE/xml/atm.xml HSfvd

Running frerun

Fremake compiled an executable for our experiment, so now we must generate a runscript that will start our experiment:

frerun -x ~/FRE/xml/atm.xml HSfvd

When frerun completes it will provide the location of the runscript. Execute this runscript to begin the simulation.

Locating your output

Once the runscript has completed, you can execute the following command to locate the output data:

frelist -d archive -x ~/FRE/xml/atm.xml HSfvd

Your run should have produced an ascii archive, a restart archive, and a history archive. Congratulations! FRE has been installed and configured.

Scientific models included with the FRE distribution

Two very simple models have been included with the FRE distribution for test purposes.

  • A finite-volume atmospheric dynamical core with idealized Held-Suarez atmospheric physics. This model is called “HSfvd” and located in xml/atm.xml. Documentation is available by requesting access to the “FMS Atmospheric Dynamical Cores” project at [[1][GForge]].
  • A simple box model of the ocean based on the MOM4p1 ocean model. This model, called “box1” and located in xml/mom4.xml, has a sector domain with a flat bottom and uses a pstar vertical coordinate. Documentation is available by requesting access to the MOM4p1 project at [[2][GForge]].

All of GFDL’s publicly released models are capable of being run with FRE, but we have not yet updated all of those public release packages to include xml. When GFDL’s publicly released models are updated over time, we will include XML in the release packages.

Sites

Each unique combination of hardware and software is designated as a site. Multiple sites may exist within a single physical site if system configurations differ sufficiently (for example: lack of a scheduler, read-only access to a filesystem, etc).

A site is defined by a number of configuration files, which are stored in a special subdirectory that is usually based on a portion of the domain name. For example, all of the Gaea machines have the domain name “ncrc.gov”, so their site directory is named “ncrc”. A sites name is defined by the site creator in the fre-system/init file.

One way to get the domain name of your machine is with the following command:

perl -T -e "use Net::Domain(hostdomain); print hostdomain . "
""

Porting FRE to a new site consists of the following steps:

  1. Install and configure the modulefiles
  2. Install the third party, prerequisite tools
  3. Install and configure fre-combiners
  4. Install and configure fre-commands, including the site configuration files
  5. Configure and run a sample experiment XML

See Configuring fre-commands for a list of files that can be customized on a per-site basis.

Site configuration examples

The FRE distribution contains a simple template site called “example”. It describes a simple site configuration with only one filesystem and only one type of CPU (no separate login, compute, i/o nodes.) You can find the site configuration files in fre-commands/site/example/. We also include two other sites for reference:

  • gfdl-ws: the site configuration files used to run simple models on GFDL’s linux workstations
  • nescc: the site configuration files used to run models on NOAA NESCC’s SGI-based high performance computer called Zeus. This is an example of a more complicated site.

Modulefiles

FRE relies on the Environment Modules system. This system dynamically changes your shell environment according to which applications you would like to have loaded via module files. The following is a simple primer on using modules:

  • module use -a /some/path: instructs the modules system to look for module files in this directory
  • module load <package>: instructs the system to load a module and make its application available to the shell
  • module unload <package>: instructs the system to unload a module and make its application unavailable to the shell
  • module list: list currently loaded modules
  • module avail: see all available modules, including which versions are set as defaults

Every FRE component and third party utility has a module file, so you can dynamically load subsets of the entire suite of applications. The modules are stored in a tree in a directory of your choosing with the application name and version as part of the tree. An example module file might look like: <install dir>/modulefiles/netcdf/4.1.1

A few module files must be edited prior to installing FRE. They are all included in the fre-system module directory: <install dir>/modulefiles/fre-system. The following is a list of environment variables that are set by the fre-system/init file.

  • FRE_SYSTEM_DIR – All compiled FRE utilities, including third party tools, will be stored here.
  • FRE_SYSTEM_DOMAIN – The domain name for the host system.
  • FRE_SYSTEM_SITE – The name of the current site.
  • FRE_SYSTEM_SITES – A colon separated list of all available sites.
  • FRE_SYSTEM_TMP – All temporary files needed by FRE will be stored here.
  • FRE_SYSTEM_ZIP – All tar files required by the FRE installer will be located here.

To configure these variables for your environment, you must add a branch to the fre-system/init file. A sample branch for domain “my.sample.com” might look like:

"my.sample.com" {
  set site sample-site
  set zipDir ~/FRE/zip
  set sysDir ~/FRE/opt
  set tmpDir ~/FRE/tmp
}

The values of the temporary variables “site”, “sysDir”, “zipDir” and “tmpDir” will be assigned to the environment variables “FRE_SYSTEM_SITE”, “FRE_SYSTEM_DIR”, “FRE_SYSTEM_ZIP” and “FRE_SYSTEM_TMP”, respectively.

You must also add your site name to the list of possible sites, defined in the “FRE_SYSTEM_SITES” variable.

Finally, the Environment Modules system must know where to look for the modulefiles, so use:

module use <path to modulefiles>

Prerequisite Software

The following software is required by FRE, and must be installed prior to FRE installation:

  • Fortran Compiler
  • C Compiler
  • Environment Modules (with Tcl)
  • Perl CPAN
  • Perl package “Date::Manip”
  • expat library
  • zlib library

The following packages are required by FRE, and are provided as part of its installation routine:

  • Perl package File::NFSLock
  • Perl package XML::Parser
  • Perl package XML::Dumper
  • Perl package XML::NamespaceSupport
  • Perl package XML::SAX
  • Perl package XML::LibXML
  • libxml2 library
  • libxslt library
  • hdf5 library
  • netCDF library (versions 3 and 4)
  • nccmp netCDF file comparison tool

We advise that you use the FRE provided packages rather than attempting to use a system default version.

Configuring fre-commands

The fre-commands files in <install dir>/opt/fre-commands/<release version>/site/<site name>/ should be modified to match your environment settings.

Here is a list of files that may need to be changed. Note that not all files are required for all sites:

  • fre.properties – Contains name-value pairs that define directory locations and other options within FRE
  • intel.mk – A makefile used by fremake for intel compilers
  • pgi.mk – A makefile used by fremake for pgi compilers
  • pathscale.mk – A makefile used by fremake for pathscale compilers
  • gnu.mk – A makefile used by fremake for gnu compilers
  • runscript.template.main – A template file that is used by the frerun to create a runscript
  • output.stager – A script to process output data
  • hsmget.inc – A make include file, used by the hsmget data copying tool
  • hsmput.inc – A make include file, used by the hsmput data copying tool

External Properties

All external properties can be found in the fre.properties file, which is located here:

<install dir>/opt/fre-commands/<version>/site/<your site>/fre.properties

The external properties file consists of lines in the following format:

name=value

The FRE convention is that all external property names must begin with “FRE.”. The file can contain empty lines and comments. Comments must begin with the “#” symbol. The “#” symbol only signifies a comment if it is provided as the first character of a line. Further, the property itself must be defined on a single line — multi-line properties are not allowed.

An example:

# --------- system defaults
FRE.directory.work.default=$TMPDIR
FRE.directory.ptmp.default=/ptmp/$USER
FRE.directory.archive.default=/archive/$USER
FRE.scheduler.prefix=#$
FRE.scheduler.runtime.max=10:00:00

The fre.properties file is read from top to bottom, and each property may refer to properties that have been defined previously — including constant properties and those defined on the command line.

All of the properties of the form “FRE.directory.*.default” can refer to the $(stem) property and all of the directory properties as well.

For example to define a post-processing directory property, where the post-processing directory resides in your archive directory, do the following:

FRE.directory.postProcess.default=$(archiveDir)/pp

You are allowed to refer only to all the previously defined directories according to the total order below:

root <- src <- exec <- scripts <- stdout <- stdoutTmp <- state <- work <- ptmp <- stmp <- archive <- postProcess <- analysis

A glossary of important properties can be found here: External Properties Glossary

Some property values may be defined as “mappings”. Mappings allow FRE to select a string based on the value of some parameter (called a “selector”).

All of the mapping elements with selectors may be omitted, then the mappings degrades to a constant value. The general mapping syntax in EBNF is below:

mapping                  = { mapping-element ";" } , default-mapping-element ;
mapping-element          = value , "{{" , Selector , "}}" ;
default-mapping-element  = value ;
selector                 = number | pattern ;
value                    = string ;

For example, the mapping below defines a maximal walltime of: 4 hours: for a number of cores less or equal to 1024 12 hours: for a number of cores between 1025 and 6000 24 hours: for a number of cores more than 6000

4:00:00{{1024}};12:00:00{{6000}};24:00:00

More complicated examples of pattern-based mappings can be found in our property files.

Conditional External Properties

The syntax of the fre.properties file allows for conditionals. That is, branches in logic can be created based on the domain of the host system.

FRE understands the following conditionals:

  • #if (expression)
  • #elsif (expression)
  • #else (expression)
  • #endif

Conditionals can’t be nested.

The excerpt below defines a property that depends on the value of the $HOST environment variable.

If this variable is equal to gaea1, gaea2, gaea3, or gaea4, then the first property will be defined, otherwise another one will be defined.

#if ($ENV{HOST} =~ m/^gaea[1-4]$/)
  FRE.scheduler.partitions=c1
#else
  FRE.scheduler.partitions=c2;t1
#endif

In the example above, FRE defines a single “c1” partition when you login at gaea[1-4], otherwise it defines two partitions (“c2” and “t1”).

Conditional external properties are implemented inside the FREExternalProperties.pm Perl module.

Compiler Makefile
Configuration

There are a number of Makefiles for holding compiler options. If you do not have a particular compiler, or do not plan to use it for compilation, you do not need to edit that compilers Makefile.

Let’s look at the gnu.mk file as an example.

This file provides four sets of compiler flags:

  • Standard flags
  • Debugging flags
  • Reproducibility flags
  • OpenMP flags

The user can choose between the above different sets of additional flags by using the target option (-t) in fremake and frerun.

If the -t option has been provided to fremake, then it will be required by the frerun command. Frerun will use this option to automatically determine the location of the executable.

Additional flags may be configured in the XML.

If you run into compilation problems, you may need to modify your library locations using the LIBS or LDFLAGS variables. The default gnu.mk file attempts to use nc-config and pkg-config to determine library locations.

Runscript Template

The runscript template file is a free-form file, written in tcsh.

This file contains a number of “pragmas”, which are expanded by frerun using information from the XML.

Examples:

  • #FRE const(name) – expands to “set -r name = value”, where the “value” depends on the given name
  • #FRE var(name) – expands to “set name = value”, where the “value” depends on the given name
  • #FRE flag(name) – expands to “set -r nameValue”, where the “Value” depends on the given name
  • #FRE version-info – expands to include the command, date, and time of the runscript creation
  • #FRE var(npes) – expands to the total number of cores for MPI processes
  • #FRE var(atmos_npes) – expands to the total number of cores for atmospheric MPI processes
  • #FRE var(atmos_nthreads) – expands to the number of atmospheric OpenMP threads per one MPI process
  • #FRE var(ocean_npes) – expands to the total number of cores for oceanic MPI processes
  • #FRE var(ocean_nthreads) – expands to the number of oceanic OpenMP threads per one MPI process #FRE var(outputDir) – expands to the directory where the final output data will be saved
  • #FRE var(gridSpec) – expands to the gridspec labeled dataFile tags from the XML
  • #FRE var(initCond) – expands to the initCond labaled dataFile tags from the XML
  • #FRE alias(runCommand) – expands to the command that was used to start MPI
  • #FRE setup-platform-csh – expands to the platform <csh> tag from the XML files
  • #FRE experiment-runtime-csh – expands to the runtime <csh> tag from the XML files
  • #FRE experiment-input-csh-init – expands to the input <csh> tag from the XML files (inserted before the main loop)
  • #FRE experiment-input-csh-always-or-postinit – expands to the input <csh> tag from the XML files (inserted inside the main loop)
  • #FRE experiment-postprocess-csh – expands to the postprocess <csh> tag from the XML files
  • #FRE dataFiles – expands to the “input” labeled dataFile tags from the XML
  • #FRE table(diag_table) – expands to hold the contents of the “diagTable” labeled dataFile tags and diagTable tags
  • #FRE table(field_table) – expands to hold the contents of the “fieldTable” labeled dataFile tags and fieldTable tags
  • #FRE table(data_table) – expands to hold the contents of the “dataTable” labeled dataFile tags and dataTable tags
  • #FRE namelists-expanded – expands to hold the contents of the expanded namelists, and should be provided inside the main runscript loop
  • #FRE namelists – expands to hold the contents of the unexpanded namelists, and should be provided before the main runscript loop, the expand_variables script will expand the name lists

The general structure of the runscript template is:

...initialization commands
@ ireload = 1
@ irun = 1
@ currentSeg = ( $ireload - 1 ) * $segmentsPerJob + $irun
...job chain state loading commands (which might modify ireload/irun)
...data loading commands

while ( $irun <= $segmentsPerJob && $currentSeg <= $segmentsPerSimulation )
  ...per-segment initialization commands
  ...call to the MPI launching utility
  ...check the MPI exit code 
  ...saving ascii files
  ...saving restart files
  ...saving history files
  ...job chain state saving commands
  @ currentSeg++
  @ irun++
end

...possible script resubmitting 
...termination commands

Your runscript template needs not conform to this structure. The frerun will perform its expansions on any runscript template it is provided.

The simplified runscript template provided with the example site is generic enough to be used without much modification on simple sites with only one filesystem and only one type of CPU (no separate login, compute, i/o nodes.) However, to keep it simple, it also doesn’t support some FRE features. For example, the runscript template for GFDL sites runs mppnccombine to combine distributed restart files. That way, the tools “frecheck” and “ardiff” can verify whether two restart files are identical, even when they have been created on a different number of processors. Without support for this feature in the runscript, frecheck and ardiff will find the files to be different.

Output Staging

FRE output staging may consist of three steps:

  • Combining: The multiple NetCDF files from the FMS output for each component are combined into a single NetCDF file.
  • Checking: An optional step in which a main run can be compared against a new run to verify consistency.
  • Archiving: The creation of the history files by creating a tar from the component NetCDF files.

FRE performs output staging in one of two ways:

  • Online: The output staging script is called directly from the runscript.
  • Staging: A staging job is submitted to the batch scheduler which handles combining, checking, archiving.

Hsmget/hsmput Configuration

You likely do not need to edit the hsmget.inc and hsmput.inc files, but internally FRE refers to two tools called hsmget/hsmput (hierarchical storage manager get and put). The settings in these files determine which copy and file manipulation commands these tools should use. The defaults are all standard unix utilities, so no changes are likely to be needed.

Some utilities that are mentioned in these files:

  • cpio
  • readlink
  • md5sum
  • tar
  • cp

These are standard Linux utilities that you likely already have.

GCP is an internal GFDL tool and is irrelevant for external sites, so the entire REMOTECP section can be ignored.

External Properties Glossary

FRE.autoRTS.archive.prefix
– archive directory to be used by the FRE “frerts” tool
FRE.autoRTS.root.prefix
– main data directory to be used by the FRE “frerts” tool
FRE.directory.analysis.default
– default value of the $(analysisDir) directory
FRE.directory.analysis.roots
– allowed roots for the $(analysisDir) directory
FRE.directory.archive.default
– default value of the $(archiveDir) directory
FRE.directory.archive.roots
– allowed roots for the $(archiveDir) directory
FRE.directory.exec.default
– default value of the $(execDir) directory
FRE.directory.exec.roots
– allowed roots for the $(execDir) directory
FRE.directory.expNamed
– a list of directories, for which the experiment name must be a part of
FRE.directory.postProcess.default
– default value of the $(postProcessDir) directory
FRE.directory.postProcess.roots
– allowed roots for the $(postProcessDir) directory
FRE.directory.ptmp.default
– default value of the $(ptmpDir) directory
FRE.directory.ptmp.paths.mapping
– a mapping to set a value of the $(ptmpDir) directory, depending on a user group id (obsolete)
FRE.directory.ptmp.roots
– allowed roots for the $(ptmpDir) directory
FRE.directory.root.default
– default value of the $(rootDir) directory
FRE.directory.root.roots
– allowed roots for the $(rootDir) directory
FRE.directory.roots.default
– allowed roots for all directories (the FRE doesn’t read this property – it’s used in the file internally)
FRE.directory.scripts.default
– default value of the $(scriptsDir) directory
FRE.directory.scripts.roots
– allowed roots for the $(scriptsDir) directory
FRE.directory.src.default
– default value of the $(srcDir) directory
FRE.directory.src.roots
– allowed roots for the $(srcDir) directory
FRE.directory.state.default
– default value of the $(stateDir) directory
FRE.directory.state.roots
– allowed roots for the $(stateDir) directory
FRE.directory.stdout.default
– default value of the $(stdoutDir) directory
FRE.directory.stdout.roots
– allowed roots for the $(stdoutDir) directory
FRE.directory.stdoutTmp.default
– default value of the $(stdoutTmpDir) directory
FRE.directory.stdoutTmp.final
– flag to prohibit user to customize the (stdoutTmpDir) directory
FRE.directory.stdoutTmp.roots
– allowed roots for the $(stdoutTmpDir) directory
FRE.directory.stmp.default
– default value of the $(stmpDir) directory
FRE.directory.stmp.roots
– allowed roots for the $(stmpDir) directory
FRE.directory.stem.default
– default value of the $(stem) property
FRE.directory.work.default
– default value of the $(workDir) directory
FRE.directory.work.roots
– allowed roots for the $(workDir) directory
FRE.expand.ARCHIVE
– the $ARCHIVE variable will be expanded by the FRE to this string
FRE.expand.HOME
– the $HOME variable will be expanded by the FRE to this string
FRE.input.staging.enabled
– flag to enable input data staging (essential on Gaea only)
FRE.mpi.runCommand.default
– default MPI run command to be used in the runscript
FRE.mpi.runCommand.solo.atmos
– MPI run command to be used in the runscript to run “solo” atmospheric models
FRE.mpi.runCommand.solo.ocean
– MPI run command to be used in the runscript to run “solo” oceanic models
FRE.mpi.runCommand.coupled
– MPI run command to be used in the runscript to run coupled models
FRE.option.dual.default
– flag to enable automatic addition the –dual option to all frerun calls
FRE.option.transfer.default
– flag to enambel automatic addition the –transfer option to all frerun calls
FRE.output.staging
– a list of possible output staging types, separated by semicolon, the first of them is the default one
FRE.output.staging.$(staging).transferOn.archive.roots
– allowed $(archiveDir) roots for the $(staging) type when transfer is on
FRE.output.staging.$(staging).transferOff.archive.roots
– allowed $(archiveDir) roots for the $(staging) type when transfer is off
FRE.scheduler.$(partition).queues
– available queues in the $(partition), separated by semicolon, the first of them is the default one (Moab)
FRE.scheduler.dual.delay
– delay (in seconds) to start a dual job
FRE.scheduler.dual.priority
– priority of a dual job (Moab)
FRE.scheduler.dual.qos
– “qos” (Quality Of Service) of a dual job (Moab)
FRE.scheduler.enabled
– flag to enable the batch scheduler
FRE.scheduler.finisher.coresSpec
– default cores specification for the finisher job (which copies the stdout from temporary to permanent location)
FRE.scheduler.finisher.partition
– default partition for the finisher job (Moab)
FRE.scheduler.finisher.queue
– default queue for the finisher job
FRE.scheduler.finisher.runtime
– default runtime for the finisher job
FRE.scheduler.inputStager.coresSpec
– default cores specification for input staging jobs
FRE.scheduler.inputStager.partition
– default partition for input staging jobs (Moab)
FRE.scheduler.inputStager.queue
– default queue for input staging jobs
FRE.scheduler.inputStager.runtime
– default runtime for input staging jobs
FRE.scheduler.jobId.length
– number of digits in the job identifier
FRE.scheduler.make.coresPerJob.inc
– minimal increment in a number of cores for make jobs
FRE.scheduler.make.coresPerJob.max
– maximal number of cores for make jobs
FRE.scheduler.make.partition
– default partition for make jobs (Moab)
FRE.scheduler.make.queue
– default queue for make jobs
FRE.scheduler.option.cpuset
– batch scheduler option pattern to enable cpusets
FRE.scheduler.option.dependsOn
– batch scheduler option pattern to define inter-job dependency
FRE.scheduler.option.generic
– batch scheduler option pattern to define generic job variables
FRE.scheduler.option.join
– batch scheduler option pattern to join the job stdout and
stderr to a single file
FRE.scheduler.option.mail
– batch scheduler option pattern to control mailing
FRE.scheduler.option.name
– batch scheduler option pattern to redefine the job name
FRE.scheduler.option.ncores
– batch scheduler option pattern to define cores number for the job
FRE.scheduler.option.partition
– batch scheduler option pattern to define partition (Moab)
FRE.scheduler.option.project
– batch scheduler option pattern to define project string
FRE.scheduler.option.projectDual
– batch scheduler option pattern to define project string for dual jobs
FRE.scheduler.option.queue
– batch scheduler option pattern to define the job queue
FRE.scheduler.option.rerun
– batch scheduler option pattern to control job reruns
FRE.scheduler.option.segmentTime
– batch scheduler option pattern to define the job segment time (obsolete)
FRE.scheduler.option.stdout
– batch scheduler option pattern to define the job stdout directory
FRE.scheduler.option.stdoutUmask
– batch scheduler option pattern to define the job stdout permissions mask
FRE.scheduler.option.time
– batch scheduler option pattern to define the job walltime
FRE.scheduler.option.workDir
– batch scheduler option pattern to define the job working directory
FRE.scheduler.outputStager.coreSpec
– default cores specification for output staging jobs
FRE.scheduler.outputStager.partition
– default partition for output staging jobs (Moab)
FRE.scheduler.outputStager.queue
– default queue for output staging jobs
FRE.scheduler.outputStager.runtime.ascii
– default walltime for ascii output staging jobs
FRE.scheduler.outputStager.runtime.restart
– default walltime for restart output staging jobs
FRE.scheduler.outputStager.runtime.history
– default walltime for history output staging jobs
FRE.scheduler.outputStager.save.coreSpec
– default cores specification for output staging saving jobs
FRE.scheduler.outputStager.save.partition
– default partition for output staging saving jobs (Moab)
FRE.scheduler.outputStager.save.queue
– default queue for output staging saving jobs
FRE.scheduler.outputStager.save.runtime.ascii
– default walltime for ascii output staging saving jobs
FRE.scheduler.outputStager.save.runtime.restart
– default walltime for restart output staging saving jobs
FRE.scheduler.outputStager.save.runtime.history
– default walltime for history output staging saving jobs
FRE.scheduler.outputStager.transfer.coreSpec
– default cores specification for output staging transfer jobs
FRE.scheduler.outputStager.transfer.partition
– default partition for output staging transfer jobs (Moab)
FRE.scheduler.outputStager.transfer.queue
– default queue for output staging transfer jobs
FRE.scheduler.outputStager.transfer.runtime.ascii
– default walltime for ascii output staging transfer jobs
FRE.scheduler.outputStager.transfer.runtime.restart
– default walltime for restart output staging transfer jobs
FRE.scheduler.outputStager.transfer.runtime.history
– default walltime for history output staging transfer jobs
FRE.scheduler.partitions
– available partitions, separated by semicolon, the first of them is the default one (Moab)
FRE.scheduler.ppStarter.coreSpec
– default cores specification for the post-processing starter job
FRE.scheduler.ppStarter.partition
– default partition for the post-processing starter job (Moab)
FRE.scheduler.ppStarter.queue
– default queue for the post-processing starter job
FRE.scheduler.ppStarter.runtime
– default walltime for the post-processing starter job
FRE.scheduler.prefix
– batch scheduler options prefix, which should be used in the header of the job script
FRE.scheduler.project
– default project name (which is used mostly for resources charging)
FRE.scheduler.queues
– available queues, separated by semicolon, the first of them is the default one
FRE.scheduler.run.coresPerJob.inc
– minimal increment in a number of cores for main runs
FRE.scheduler.run.coresPerJob.max
– maximal number of cores for main runs
FRE.scheduler.run.partition
– default partition for main runs (Moab)
FRE.scheduler.run.queue
– default queue for main runs
FRE.scheduler.runtime.max
– maximal walltime (might be defined as a mapping depending on a number of cores)
FRE.scheduler.submit.command
– batch scheduler submit command
FRE.scheduler.submit.output.pattern
– expected pattern for the batch scheduler submit command response
FRE.scheduler.stdout.temporary.glob
– a glob to locate the temporary stdout file with a given job identifier
FRE.scheduler.variable.environment
– name of an environment variable to determine the job type (batch or interactive)
FRE.scheduler.variable.environment.value.batch
– value of an environment variable above, identifying a batch job
FRE.scheduler.variable.jobID
– name of an environment variable, containing a job identifier
FRE.scheduler.workDirCleaner.coreSpec
– default cores specification for the work directory cleaning job
FRE.scheduler.workDirCleaner.partition
– default partition for the work directory cleaning job (Moab)
FRE.scheduler.workDirCleaner.queue
– default queue for the work directory cleaning job
FRE.scheduler.workDirCleaner.runtime
– default walltime for the work directory cleaning job
FRE.tool.archiver.extension
– an extension to be used by archiver (for example – "tar")
FRE.tool.fremake.enabled
– a flag to enable the fremake tool
FRE.tool.frepriority.enabled
– a flag to enable the frepriority tool
FRE.tool.frerun.enabled
– a flag to enable the frerun tool
FRE.tool.make.jobs.default
– default number of children processes in a make job
FRE.tool.make.override.netcdf.mapping
– a mapping to set a make NETCDF override, corresponding to a netCDF library version present in the user platform
FRE.tool.mkmf.template.default
– default mkmf template
FRE.tool.mkmf.template.mapping
– a mapping to set a mkmf template, corresponding to a compiler present in the user platform
FRE.tool.modules.home
– the home directory for environment modules
FRE.tool.modules.use
– the directory with FRE modulefiles (to be used in the "module use" command)
FRE.versioncontrol.enabled
– a flag to enable access to a version control system
FRE.versioncontrol.cvs.root
– default root of the CVS repository (might be remote)