GFDL - Geophysical Fluid Dynamics Laboratory

Using FRE on Gaea

FRE is available on Gaea and the TDS (the Test and Development System). This page describes the status of FRE development and how to use it. The latest release of FRE is arkansas-10.

Quickstart Guide

Get started fast with this example of checking out XML files, compiling an executable, running a regression test, and running a production run — including launching postprocessing at GFDL. This model takes about an hour per model year for the production run, and will run seven years so that you can see it reload a few times. Postprocessing for this model skips the first year and then will be done in 3 and 6 year increments.

1. Log onto Gaea from GFDL once your account has been set up.

ssh gaea

2. On Gaea, load FRE, check out xml, and launch fremake.

module load fre
mkdir gaea_quickstart; cd gaea_quickstart
cvs co -r arkansas-10 xml
cd xml
frelist -x quickstart.xml
fremake -s -t openmp -x quickstart.xml m45_am2p14_1990
### or run without "-s" and run the compile script directly

3. Monitor the jobs in the Moab scheduler. The second invocation is more verbose: it shows job names as well as job numbers.

showq -u $USER
showq -u $USER -n -v

4. While that compiles, use the predefined executable in the child experiment to launch runs. If you prefer to use the executable you just created, wait until the compile is finished and remove line 1082, the <executable file=""> line, before proceeding with the commands below. You can run the experiment “m45_am2p14_1990”, but it will be much slower.

### regression run suite
frerun -r suite --submit-staged -t openmp -x quickstart.xml m45_am2p14_1990_3thread

### production run
frerun  --submit-staged -t openmp -x quickstart.xml m45_am2p14_1990_3thread

5. Locate the files being generated during the run.

frelist -dir -t openmp -x quickstart.xml m45_am2p14_1990_3thread

### stdout of in-progress jobs is in the directory where the job was launched
ls *.OU

6. Check the compile after the compile job finishes.

frestatus -t openmp -x quickstart.xml m45_am2p14_1990

7. Check the results of the regression runs.

frecheck -t openmp -x quickstart.xml m45_am2p14_1990_3thread

8. Check the results of the production run when it is complete. Look for the output at GFDL’s archive filesysem in: $ARCHIVE/gaea_quickstart/. The xml file will be sent to the corresponding directory to its Gaea location. The freppcheck command should be run at GFDL.

cd $ARCHIVE/gaea_quickstart/xml
freppcheck -t 1983 -y 1988 -x quickstart.xml m45_am2p14_1990_3thread

State of FRE Development on Gaea.

End-to-end runs consisting of the following have been shown to work:

  • code check-out and compilation at Gaea,
  • input data staging in the ltfs filesystem,
  • production runs that resubmit themselves and have dual runs,
  • history data combined automatically,
  • data sent to the /archive filesystem, and
  • post-processing automatically launched.

Specifically, the FRE tools fremake, frerun, frecheck, and frelist have been shown to work.

Input data staging is in available from the long-term fast scratch (ltfs) to the fast scratch (fs) filesystems. To use data not on the fs filesystem, you should use the data staging frerun options; see below. Use of data directly from GFDL’s /archive filesystem is available and undergoing testing.

FRE has a history data staging job to combine history files. The runscript and history data staging job launch data transfer jobs to send data back to GFDL. They require valid grid proxy certificates. Submitting post-processing jobs at GFDL from the Gaea runscripts works. There is one issue in that there is no mechanism to enforce frepp jobs be called in order when jobs are submitted out of order due to data transfer issues. A solution for this problem is in development.

Dual runs occur by default with all production runs. You can turn this off with the frerun –nodual option.

To use FRE, add it to your environment.

This command is already done in your account setup to make FRE modules available, but note that it must also in the platform section of your xml.

module use -a /ncrc/home2/fms/local/modulefiles

To load FRE into your environment, do

module load fre

The default and recommended version of FRE on Gaea is module load fre/arkansas-10, which is what you will currently get with module load fre.

Input Data Staging

Input data staging is needed in order to transfer needed files from LTFS to FS prior to a run. The batch nodes do not have LTFS mounted. This means jobs needing access to files on LTFS will not work properly. In order to handle this there are two available modes of input data staging. Once the data staging script has run successfully, as long as the relevant paths still exist, runscripts can be submitted many times without requiring another input data staging script.

(1) frerun option –submit-chained

The “submit-chained” mode works by having frerun create the normal runscript. This runscript first initializes an input data staging job. That job loads input data into the “ptmp” directory before it submits the regular runscript as the second job. Once the data staging is complete, another job is started which is the actual job itself. This means submit-chained only creates one job at a time. The command for this is:

 msub -N <name> -j oe -l size=1,walltime=<time>,partition=es -q eslogin -v FRE_STAGE=CHAIN <script>

A useful csh/tcsh alias for submitting a chained job:

alias msub_chain 'msub -N !:1_chain -j oe -l size=1,walltime=01:00:00,partition=es -q eslogin -v FRE_STAGE=CHAIN !:1'

Then submit the chained job by running:

msub_chain <script>

(2) frerun option –submit-staged

The “submit-staged” method works by having frerun create the normal runscript and submit it twice. Two jobs are submitted at once. The first job contains a submission with options so that it is forced to work in “staging mode”. This will cause the copy over from LTFS to FS. The second job is the actual runscript job. The second job will wait in the blocked queue until the first job completes. Virtually, your job cannot not run until all the necessary files are transfered to the FS filesystem. When they are, job 1 will complete and the run job, job 2, will start. The command for this is:

msub -N <name> -j oe -l size=1,walltime=<time>,partition=es -q eslogin -v FRE_STAGE=INPUT <script>

The second runscript is submitted with options which force it to wait for the first job to finish. This means the <jobId> reported by the first msub submission must be used in the second command:

msub -l depend=afterok:<jobId> <script>

<name> can be any name for input staging job. This is not mandatory. The name is set in the runscript but this declaration will allow users to name the staging job differently so its output file will also have a different name.

<time> is its walltime

<script> is regular runscript </pre>

A useful csh/tcsh alias for submitting a staged job:

alias msub_stage 'msub -N !:1_stage -j oe -l size=1,walltime=01:00:00,partition=es -q eslogin -v FRE_STAGE=INPUT !:1 | xargs -I{} msub -l depend=afterok:{} !:1'

Then submit the stage job by running:

msub_stage <script>

How to update your XML files for Gaea

Sample XML files.

The next FMS release will contain XML files updated for Gaea, but for now, even Modeling Services is still working on our XML files. The best way to get the latest xml file is to contact your Modeling Services liaison. You can find some recent examples with

cvs co -r arkansas-10 xml 

System entities are likely to cause a problem

Note that system entities refer to a file on the system. If this file doesn’t exist, you will get errors from any xml tool. You may need to work around this by having a “dummy” set of entities. For example:

<!DOCTYPE experimentSuite [
<!ENTITY analysis_am_atmos_av_annual_20yr SYSTEM "/home/atw/fms/diags/xml_entities/analysis/am/atmos/av/annual/20yr.xml">
]>

<!-- To run at ORNL, comment the DOCTYPE above and uncomment this one: -->
<!DOCTYPE experimentSuite [
<!ENTITY analysis_am_atmos_av_annual_20yr   "">
]>

Controlling directory locations with the FRE directory stem.

We’ve made the default directory locations the best known practices at this time. Using them is recommended. We’ve also added a new directory customization called the “stem” — it’s the partial directory path to be used with the default directory locations, to describe where you want to store all your files on the different filesystems. This feature makes it easier to work with directory locations between the two sites. Here’s an example:

<experimentSuite rtsVersion="4">

<property name="FRE_STEM" value="riga_201011"/>

<setup>

<platform name="hpcs.default">

<directory stem="$(FRE_STEM)"/> ...

Each platform needs only that one directory line. Then with “frelist -dir”, which lists directory paths, you can see the file locations.

The example gives the following default directories at Gaea:

frelist -dir -x CM2.1U.xml CM2.1U_Control-1990_E1.M_3A

root: /lustre/ltfs/scratch/Amy.Langenhorst/riga_201011
src: /lustre/ltfs/scratch/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/src
exec: /lustre/ltfs/scratch/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/ncrc.default-prod/exec
scripts: /ncrc/home1/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/ncrc.default-prod/scripts
stdout: /lustre/fs/scratch/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/ncrc.default-prod/stdout
work: /lustre/fs/scratch/Amy.Langenhorst/work/$PBS_JOBID
ptmp: /lustre/fs/scratch/Amy.Langenhorst/ptmp
archive: /lustre/fs/scratch/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/ncrc.default-prod
postProcess: /lustre/fs/scratch/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/ncrc.default-prod/pp
analysis: /lustre/fs/scratch/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/ncrc.default-prod/analysis

and on the new analysis cluster at GFDL, it gives:

frelist -dir -x CM2.1U.xml CM2.1U_Control-1990_E1.M_3A

root: /home/Amy.Langenhorst/riga_201011
src: /home/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/src
exec: /home/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod/exec
scripts: /home/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod/scripts
stdout: /home/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod/stdout
work: $TMPDIR/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod
ptmp: $TMPDIR/ptmp
archive: /archive/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod
postProcess: /archive/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod/pp
analysis: /archive/Amy.Langenhorst/riga_201011/CM2.1U_Control-1990_E1.M_3A/gfdl.default-prod/analysis

Don’t forget that you need to provide frelist with the “-t target” option for openmp or other non-default targets.

If you want to use a FRE stem but need to customize a directory, you can use xml like this:

<directory stem="$(FRE_STEM)">

<src>$HOME/$(stem)/src</src>

</directory>

Add the ncrc platform(s) you’d like to use to your XML file.

The intel compiler is generally the recommended compiler for FMS-based codes for efficiency reasons and known issues with PGI and Pathscale. However, Pathscale has better behavior and efficiency with openmp. Please consult your Modeling Services liaisons for the best known configuration for your model.

<platform name="ncrc.default">

<directory stem="$(FRE_STEM)"/>

<csh><![CDATA[

source $MODULESHOME/init/csh

module use -a /ncrc/home2/fms/local/modulefiles

module unload PrgEnv-pgi PrgEnv-pathscale PrgEnv-intel PrgEnv-gnu PrgEnv-cray

module unload netcdf fre

module load PrgEnv-intel/3.1.29

module swap intel intel/11.1.073

module load fre/arkansas-10

module list

setenv MPICH_MAX_SHORT_MSG_SIZE 8000

setenv OMP_NUM_THREADS 1

setenv PSC_OMP_AFFINITY FALSE

setenv NC_BLKSZ 1M

setenv F_UFMTENDIAN big

]]>


</csh>

<property name="SGI_FLAGS" value="" />

<property name="INTERNAL_NML_FLAG" value="-DINTERNAL_FILE_NML" />

<property name="FMS_ARCHIVE_ROOT" value="/lustre/fs/archive/fms" />

</platform>



<platform name="ncrc.pathscale">

<directory stem="$(FRE_STEM)"/>

<csh><![CDATA[

source $MODULESHOME/init/csh

module use -a /ncrc/home2/fms/local/modulefiles

module unload PrgEnv-pgi PrgEnv-pathscale PrgEnv-intel PrgEnv-gnu PrgEnv-cray

module unload netcdf fre

module load PrgEnv-pathscale/3.1.29

module load fre/arkansas-10

module list

setenv MAIN_PROGRAM coupler_main.o

setenv MPICH_MAX_SHORT_MSG_SIZE 8000

setenv OMP_NUM_THREADS 1

setenv PSC_OMP_AFFINITY FALSE

setenv NC_BLKSZ 1M

]]>


</csh>

<property name="SGI_FLAGS" value="" />

<property name="INTERNAL_NML_FLAG" value="" />

<property name="FMS_ARCHIVE_ROOT" value="/lustre/fs/archive/fms" />

</platform>

Setting the CVS repository.

There is a new CVS server that lies between GFDL and Gaea on the network. This server has access to GFDL’s home filesystem and can serve the CVS repository directly to Gaea. It can be used for both CVS check-out and check-in. FRE sets up the following for you, but if used independently of FRE, you would need the following settings:

  setenv CVS_RSH gsissh
  setenv CVSROOT :ext:cvs.princeton.rdhpcs.noaa.gov:/home/fms/cvs

This is certificate-based rather than ssh tunnel-based, so it works interactively and in batch scripts.

Specifying data files at multiple sites.

A useful method of specifying data files is to specify one or more root locations in the platform tag with properties.

<platform name="ncrc.default">

<property name="FMS_ARCHIVE_ROOT" value="/lustre/fs/archive/fms" />

...

Then the dataSource can make use of these root locations.

<dataFile label="input" target="INPUT/" chksum="" size="" timestamp="">

<dataSource platform="$(platform)">$(FMS_ARCHIVE_ROOT)/module_data/quebec/navy_topography.data.nc</dataSource>

</dataFile>

This is equivalent to

<dataFile label="input" target="INPUT/" chksum="" size="" timestamp="">

$(FMS_ARCHIVE_ROOT)/module_data/quebec/navy_topography.data.nc

</dataFile>

This is also equivalent to the following, where hpcs, ncrc, and doe are all available sites known to FRE. This syntax allows you to customize input files by site if necessary.

<dataFile label="input" target="INPUT/" chksum="" size="" timestamp="">

<dataSource site="hpcs">$(FMS_ARCHIVE_ROOT)/module_data/quebec/navy_topography.data.nc</dataSource>

<dataSource site="ncrc">$(FMS_ARCHIVE_ROOT)/module_data/quebec/navy_topography.data.nc</dataSource>

<dataSource site="doe">$(FMS_ARCHIVE_ROOT)/module_data/quebec/navy_topography.data.nc</dataSource>

</dataFile>

Note that a subset of /archive/gold is currently synced to /lustre/fs/archive/fms/gold. A subset of /archive/cm3 directory is synced to /lustre/fs/archive/fms/cm3. If there is data missing from /lustre/fs/archive/fms, let Amy know and she will sync it.

Using OpenMP .

To use OpenMP the command line which runs the model needs to be customized. The runscripts are now created as:

  alias runCommand `which aprun` -n $npes ./$executable:t
...
  runCommand

So as an example, one could customize the run command by resetting the alias:

<csh>

setenv mem_per_thread 2

alias runCommand "aprun -m $mem_per_thread -d 6 -n 30 ./$executable:t"

</csh>

Please see the XML files for more examples.

Post-processing on the New Analysis Nodes

To postprocess automatically from Gaea and to send frepp jobs to moab on the new pp/an nodes, you need a platform “gfdl.default” in your xml files. Copy your hpcs.default platform to a new platform called “gfdl.default”. (Or prefix your other platforms with “gfdl.”.) Use platform csh like this:

       source $MODULESHOME/init/csh
       module use -a /home/fms/local/modulefiles
       module purge
       module load fre/arkansas-10

To call frepp by hand, log onto the analysis nodes and do

       module use -a /home/fms/local/modulefiles
       module load fre/arkansas-10
       frepp...

Here is an example gfdl.default platform:

<platform name="gfdl.default">

<directory stem="$(FRE_STEM)"/>

<csh><![CDATA[

source $MODULESHOME/init/csh

module use -a /home/fms/local/modulefiles

module purge

module load fre/test

]]>
</csh>

<project>gamd</project>

<property name="FMS_ARCHIVE_ROOT" value="/archive/fms"/>

<property name="SGI_FLAGS" value=""/>

</platform>

Tips and Troubleshooting

Help! My batch scripts cannot read or write to /lustre/ltfs !

For efficiency reasons, batch nodes can only see the /lustre/fs filesystem and the home filesystem. Soon FRE will support staging data from /lustre/ltfs and /archive, but for now, your input data and archivedir, along with workdir and ptmpdir, must be on /lustre/fs.

Help! How do I specify reference restart files for different sites?

Please follow this example:

<dataFile label="reference">

<dataSource site="hpcs">/archive/pjp/.../00000008.cpio</dataSource>

<dataSource site="ncrc">/lustre/fs/scratch/Amy.Langenhorst/.../00008.tar</dataSource>

</dataFile>

The HPCS_PLATFORM and NCRC_PLATFORM properties must be defined in the platform section of the XML as shown above.

Help! My XML file uses ncvarrename and that program doesn’t exist!

The FRE environment module loads the NCO utilities: ncap, ncap2, ncatted, ncbo, ncdiff, ncea, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncrename, ncwa, and udunits2.

To locally rename a variable in a csh section in the model’s INPUT directory, please use:

mv restore_mask.nc foo
ncrename -v tmask,restore_mask foo restore_mask.nc

The “mv” command is necessary to “release” the file from ptmp dir, otherwise you modify the copy in ptmp dir and later use of that file may have incorrect behavior.

Help! Pathscale tried to use the wrong main program.

You can tell Pathscale what the main program should be by setting the MAIN_PROGRAM environment variable. For coupled models, this should be set to “coupler_main.o”. For solo models, it needs to be set to the solo model’s main program.

<platform name="ncrc.pathscale">

<csh> <![CDATA[

setenv MAIN_PROGRAM atmos_model.o

]]>
</csh>

Help! Batch jobs fail to send data to GFDL.

Please make sure the $X509_USER_PROXY is set in your login session. If it is not, you may need to adjust your .cshrc. Please see Configure the Gaea Connection

Please also make sure you are using:

<postProcess combine="staged"/> 

Help! How can I easily send a file back to GFDL? And from /archive to Gaea?

FRE includes a tool called “send_file” which sends a file from Gaea filesystems to GFDL’s /archive filesystem. The FRE runscripts use this to send the model data to GFDL. When used interactively, this script is intended to be run from the Gaea login nodes. The archive path will be constructed from the Gaea path as follows. The Gaea paths:

/lustre/ltfs/scratch/username
/lustre/ltfs/stage/username
/ncrc/home*/username

will be translated to

/archive/$USER

and the remaining portion of the filename will be replicated. For example,

send_file /lustre/ltfs/scratch/user1/fms/testing/fileA

will submit a batch job to send the file to

/archive/$USER/fms/testing/fileA

This tool can be run interactively, so that it does not submit a batch job. Use the -i option to run the transfer interactively.

send_file -i /lustre/ltfs/scratch/user1/fms/testing/fileA

This tool can also be submitted as a batch job directly to Moab via

msub -v file=/lustre/ltfs/scratch/user1/fms/testing/fileA `which send_file`

Also, an analogous script “get_file” to transfer data from the GFDL /archive filesystem to the Gaea ltfs filesystem is available.

For example,

get_file /archive/username/fms/testing/fileA

will move the file to

/lustre/ltfs/scratch/$USER/fms/testing/fileA

Run “send_file -h” and “get_file -h” at Gaea to see help messages.

Help! Where’s my stdout file during the run?

During the job run, the stdout is in a temporary file in the directory from which the msub was launched. After the run, it is moved to the location that was specified by FRE or in the msub command.

Help! How do I get an interactive session?

Use “msub -I” to get an interactive session. With no other arguments, this will give you 24 cores for 1 hour on the main Gaea compute resources. This is the minimum amount of cores you can request. To request more resources, use:

msub -I -l size=48 -l walltime=02:00:00

Help! How do I use the Moab scheduler?

See the MoabTips page for useful Moab commands.

Help! How do I use the transfer utility gcp?

See GfdlCP.

Help! How do I use hsmget and hsmput?

See HsmgetAndHsmput.

Help! My mppnccombine/fresavehist jobs failed, and so my history data is uncombined and the history tar file didn’t get created correctly.

You can resubmit the job to do the mppnccombines as follows:

setenv datedir [dir containing files to be combined]
setenv stdout [dir where you want the stdout to go]
msub -l size=48,walltime=4:00:00,partition=c1ms -v datedir,stdout /ncrc/home2/fms/local/opt/fre-commands/arkansas-10/site/ncrc/fresavehist

If you want it to also launch the frepp command at gfdl, it would be

setenv datedir [dir containing files to be combined]
setenv stdout [dir where you want the stdout to go]
setenv frepp [the frepp command]
msub -l size=48,walltime=4:00:00,partition=c1ms -v datedir,stdout,frepp /ncrc/home2/fms/local/opt/fre-commands/arkansas-10/site/ncrc/fresavehist

where the frepp command looks like

frepp -t 19810101 -d /lustre/fs/scratch/.../history -x /ncrc/home1/..../file.xml exptname

fresavehist will convert the frepp command as needed to be used at gfdl. This works on history directories where some of the combines succeeded and some failed.

Help! CVS commands print the system information message, and I don’t want to see it!

In your ~/.ssh/config on Gaea, “LogLevel ERROR” will improve this:

Host gfdl
  HostName localhost
  LogLevel ERROR

Help! Compiles in batch fail, but work interactively!

There was a bug with Moab that has been fixed. If you see this problem, please let Frank Indiviglio know immediately.

Help! When I msub a job, I get a “corrupt XML” error message. What’s wrong with my XML?

Nothing is wrong with your XML — This was a bug with Moab that has been fixed. If you see this problem, please let Frank Indiviglio know immediately. Moab actually converts the jobs it schedules to XML internally in the background, and hit an internal error.

Help! My question is not listed here.

Please submit a help desk ticket. Please include the following information within that ticket:

  • Hostname (eg. Gaea2)
  • Time of occurrence
  • Date of occurrence
  • Path to necessary files or scripts
  • Details on the issue occurring