FRE Version History

Allows mkmf template to be specified in the xml itself.

<mkmfTemplate file=""/>
or
<mkmfTemplate>
 FC = f90
 CPPFLAGS = -macro_expand
 etc
</mkmfTemplate>

This can go in the <setup> tag, or the setup tag can be overridden if you specify the mkmfTemplate in the <compile> section for your experiment.

Warning

This requires a change in syntax from previous specifications of the mkmf template. One-word mkmf templates will cause fremake023 to print an error message, ie, <mkmfTemplate> /home/fms/bin/mkmf.template.ia64_flt </mkmfTemplate> yields ERROR: mkmfTemplate '/home/fms/bin/mkmf.template.ia64_flt' looks like a file. ERROR: You must use the file attribute: <mkmfTemplate file=''> you would need to change it to <mkmfTemplate file="/home/fms/bin/mkmf.template.ia64_flt"/>

When your machine is upgraded to the new OS, CVS will be updated, and it will no longer generate the path_names file automatically. However, if you use FRE for checking out code from the repository, fremake will generate your path_names file for you automatically, just after it executes your CVS commands. If you use fremake for acquiring and compiling code, the CVS upgrade should be invisible to you.

In your xml, you can now use two <mkmfTemplate> tags:

   <mkmfTemplate file="/home/vb/fms/mkmf.template.ia64_flt"/>
   <mkmfTemplate type="debug" file="/home/vb/fms/mkmf.debugtemplate.ia64_flt"/>

Then if you use the -t option, fremake will always use the "debug" mkmf template if one is listed in your xml. If one is not listed, it'll use the system default (/home/fms/bin/mkmf.debugtemplate.$platform). If you don't use -t, it'll use the non-debug template.

fremake will put a comment line at the top of any mkmf template it acquires from an external file, ie

#from /home/vb/fms/mkmf.template.ia64_flt

new version of mkmf allows for calling mkmf from exec dir instead of src dir. fremake uses this to allow you to compile the optimized and debug versions at the same time.

new option -p to call list_paths only once, after all the cvs stuff is complete. This is much faster. This works for all experiments except mom4 ones that treat the mom4 paths differently than the shared code paths.

Now defaults to call mppnccombine to combine history files within the model runscript, not in a separate job between the production run and postprocessing jobs. To combine history files in a separate job, use

     <postProcess combine="offline">

This change was made on 20081031 due to issues with offline combining of history data and regression runs.

Added test for existence of archive after the run, so that it will cleanly resubmit itself if archive is not available.

Modified to determine whether to call mppnccombine or the land model ncc-combine program on distributed restarts when it is necessary to combine them.

Instead of always being saved with '.0000.' in the filename, now only distributed restart files will have that in the filename.

Some regular expressions modified to support ensemble filenames.

Modified to support multiprocessor ascii out files in the future, when they are implemented in FMS.

Modified to print one data file per line on the command line arguments to get_fms_data, for easier comparison of runscripts.

Implemented the ability to override any namelist within the regression run tag. This makes it possible to override atmos_npes as well as atmos_layout, for example. The syntax is:

<regression>
  <run ... overrideParams="namelist:var=val;namelist:var=val;..." />
ie,
<regression>
  <run ... overrideParams="fv_core_nml:atmos_layout=3,30;fv_core_nml:atmos_npes=90;"/>

This replaces the functionality in the existing frerun to override the layout parameter in various namelists. Backward compatibility is preserved; it just prints a warning message to change your xml. Algorithm:

if the namelist has been specified,
  if the variable has been specified, replace it
  if the variable has not been specifed, add it
if the namelist has not been specified, add the namelist and the variable

This feature is ONLY available in FRE3, not in FRE2.

Calls frepp025_interp

mppnccombine updated: mppnccombine-2.1.7_ia64

get_fms_data updated: cpio now checks file dates when extracting, will overwrite older files that exist with newer files when two cpio files are extracted to INPUT

emailing the help desk on production job crash no longer mails whole stdout file

will look for a user checkpoint file in the user's home directory

remove *.res.nc as well as *.res from INPUT when setting up TMPDIR between restarts (seems to be a bugfix, I'm not sure if it was affecting anyone) (v2/frerun025 only, not applicable to FRE3)

bugfix to making a cpio after combining distributed restart files, for regression runs only (v3/checked out frerun only)

Calls frepp025

Supports Bernie Sieber's history data staging. To try it, you must use the following in your xml file: <postProcess combine="staged">

Calls frepp024_64_vb

does not combine restart files; therefore this version supports distributed restart files. NOTE: This may be incompatible with current versions of the land model. More testing is required with the land model.

defaults to do mppnccombine on history files in a separate job in between the production run and postprocessing jobs. This is changed from the previous default of combining history files in production jobs. To combine history files in the production jobs in this version, use <postProcess combine="online">

creates new 64-bit netcdf format to support large data files

uses /home/fms/bin/mppnccombine-2.1.4_ia64 (with -64 option)

uses /home/fms/bin/get_fms_data_cxfscp

calls frepp024_64

uses "exit 99" for checkpoints to improve checkpointing behavior

formats all namelists in scripts with a newline before the end '/'

uses qsub soft resources to tell the operators job statistics

Calls frepp024_99

ability to extend a run with -e

you can provide a cpio file as a grid spec and frerun knows that it should be extracted rather than just copied to the $work dir.

Calls frepp024_grid

New option -l: frerun passes the argument on to "qsub -l". Use this to submit to a certain node, or any other qsub -l resource, ie, frerun -l ic1 ...

System monitoring commands added: times the mpirun command, calls /home/gcs/bin/cpusetmeminfo before each mpirun command, to print a list of free and used memory on the nodes of the current cpuset.

Checkpointing functionality added: If /home/gfdl/flags/fre.checkpoint.* have been created by OPS, the runscript will resubmit itself and exit. The runscript checks for these flags before every mpirun command. Mail will be sent to the user when a job has been checkpointed.

Saves a copy of the xml file into each restart cpio file.

Uses Hans's newest /home/fms/bin/mppnccombine-2.1.1_ia64

Archives a copy of the batch stdout for jobs that failed for the Technical Services group's reference

Calls frepp024.

If you are running an experiment using the FMS coupler, frerun will look to make sure your fieldTable includes sphum for both land and atmos. If not, it will add them.

Adds an extra return character before the '/' ending a namelist. This allows Fortran comments on the last line of a namelist specification in the xml.

Cpios data files directly to and from archive with a buffer size of 512K, instead of creating the cpio in $TMPDIR and using cp/mv to move the file to archive. This should result in a speed improvement, because cp has a small buffer size on the altix platform which slows down file transfers.

Calls frepp023_timavg

calls frepp023

Now overwrites the fv_core_nml and land_model_nml's layout parameters correctly if you specify the atmos_layout and land_layout attributes in your regression test xml.

Fills in the $baseDate variable correctly if used in namelists

It is now possible to create a runscript with neither MPI nor SHMEM.

creates a runscript which will perform mppnccombine inline in the job script rather than calling frepp to do the combine. This will save one round of queue wait times for the model postprocessing. The 'raw' (uncombined) history files will not be saved to archive.

since mppnccombine is online, never call frepp for regression test runs

calls frepp022 for postprocessing

writes to ~/.rhosts file if necessary for file permissions

small improvements to mppnccombine and dmgets in the runscript

calls frepp021

new cshell postInit option. There are four categories of csh you can use within the <input> tag:

      <csh>
#cshell to be run from INPUT/ at the beginning of each job
      </csh>
      <csh type="always">
#cshell to be run from INPUT/ before each mpirun command  
      </csh>
      <csh type="init">
#cshell to be run from INPUT/ before the very first mpirun command of the
#first job of the experiment only
      </csh>
      <csh type="postInit">
#cshell to be run from INPUT/ before each mpirun command EXCEPT the very
#first mpirun command of the experiment
      </csh>

namelists in runscript will be sorted alphabetically

new attribute zetac_layout is processed

modified subroutine getexecutable to always check whether an inherited experiment should have its own executable or not

bugfix to production runs of less than 1 year; will now set an appropriate combineFreq for these runs

new option for reading all the namelists for an experiment and outputting all namelists alphabetically as text in xml tags.

bugfix to procedure of finding the debug executable for inherited experiments

changed so that project tag gets placed in regression runs as well as production runs

ability to run on altix platform

prints a warning if make_exchange_reproduce is false for regression tests, and prints a warning if make_exchange_reproduce is true for production runs.

added code to combine multithreaded write restart files

added code to change priority projects to dev projects when submitting a windf job

just call the corresponding version of the postprocessing program

better wording on namelist warnings

supports platform csh for changing version of compiler. To use the new version of the compiler, use xml as follows:

<setup>
   <target platform="sgi">
      <csh>
        source /opt/modules/modules/init/csh
        module switch mpt mpt_1900
        module switch mipspro mipspro_741m
      </csh>
   </target>
</setup>

and remove the old <targetPlatform>sgi</targetPlatform>.

multiple types of input csh supported

<input>
  <csh> this is executed in the section which copies input files at the beginning
        of each JOB, same as previously
  </csh
  <csh type="init"> this is executed in the section which copies input files ONLY ONCE
        PER EXPERIMENT, only at the initial run time t=0
  </csh>
  <csh type="always"> this is executed in the mpirun loop so that it is executed
        BEFORE EVERY MPIRUN command
  </csh>

developmental changes, calls rtspp9

Doesn't attempt to compile an executable if you specified an existing one in your xml

can now set project for jobs with rtspriority

Modified to use new version of fregrid's feature to set 1st order interpolation as a netCDF attribute. The following variables are now set to interpolate 1st order:

precip prec_ls snow_ls prec_conv snow_conv uw_precip prec_uwc snow_uwc prc_deep_donner snow_deep_donner prc1_deep_donner prc_mca_donner snow_mca_donner ice_mask land_mask zsurf 

Other variables will be interpolated to 2nd order.

Fixed a variable scope-related issue that caused a frepp crash when the user requested an annual timeSeries with a smaller chunk length than the monthly timeSeries.

mppnccombine updated: mppnccombine-2.1.7_ia64

replaced "dmput -r" with "dmput" to prevent having to wait for postprocessed files in analysis jobs

will look for a user checkpoint file in the user's home directory

updated handling of 1-D static variables with fregrid

uses fregrid instead of Mike Herzog's tool for converting cube sphere grids to lat-lon.

supports (and always creates) 64-bit netcdf format files

jobs use 4 processors. If jobs are creating a 20-year or longer time series or time average, for any component, they use 8 processors.

if the time-averaging program receives an out-of-memory kill error, the command will be retried. If you are still getting a lot of post-processing errors with this version, and you see a lot of results from "grep Killed $root/scripts/postProcess/stdoutfile", please let Amy know.

updated analysis figure code from Andrew Wittenberg and Fanrong Zeng

supports cube sphere grid with Michael Herzog's interpolation tool.

New copy utility, cxfscp, to copy data from archive to TMPDIR for processing. This is about 2x to 3x faster than previous methods of cpio from archive or using "cp", and has resulted in a 25% speedup in most post-processing tests. Performance benefits will vary with system load

New external utilities:

Correct histDir is passed to analysis scripts that need it (bugfix to case where you want to create figures from someone else's postprocessed files and still require the history data.)

Improved method for calculating required vars for converting to pressure levels.

Added checkpointing with saving status to a file, similar to reload_commands for model runscripts. The script writes its status to $ppRootDir/.checkpoint/ as it progresses through a component, and deletes that file when the job is finished. At the beginning of the job script, it checks for the existence of that file and goes to the indicated position in the script.

New option -l: limit static variable processing to the diagnostic source file listed in the postprocessing tags of your xml. Otherwise, frepp will try, for example, to put all atmos-related static variables into post-processing component static files with a "type" string starting with "atmos".

Doesn't attempt to make the "pp/" directory level with frepp -A.

Gives an error message when the user-specified pp directory does not exist when frepp is called with -A (analysis-only.)

Has less verbose stdout messages. To obtain more verbose output, use the "frepp -v" option.

can generate an annual time series directly from history files (without the intermediate step of monthly data) when the diagnostic file involved has "year" as the third column in the diag table file specification.

improved method of determining which variables are required for interpolation to pressure levels

increased cpio buffer size on altix

the altix platform is the default platform for post-processing. To force post-processing on the SGI Origins with this version of frepp, you need to specify the post-processing platform as "sgi as follows:

<postProcess platform="sgi">
...

All versions of frepp023* will be changed to this setting on 10/24/2006.

Utilities used by FRE have been recompiled for the Altix. They have -fltconsistency and all the rest of the default fms compiler options. This should result in increased performance.

Bugfix on labeling the time axis of 100yr timeAverages (or any timeAverage of timeAveraged files, where the number of files being averaged is odd, as in the 100yr case.)

Let jobs covering 1 year of postprocessing have 2 hours of cpu time instead of 1 hour, as hi-res models need this extra time.

Additional atmospheric plevel interpolation preset: narcaap

Can perform interpolation to pressure levels when temp, sphum, and zsurf are missing (just can't calculate divv, rvort, hght, slp without them.)

If this job has no history data files, do not run a blank "dmget $historyfiles" line.

new -R option to force regeneration of analysis figures

use FFIO layer for all netcdf utilities EXCEPT ncks for best efficiency

use Remik's suggestions for NETCDF_FFIOSPEC for best efficiency

use Bernie's mvfile utilities for safe and preallocating file moves from $TMPDIR to /archive

use ncrcat instead of nccatm for speed

submit timeAverages in a separate job from timeSeries if component is ocean and there will be calculates on the 100 year boundary or greater

if the component is ocean and there will be calculations of more than 20 years of data, request 30 cpu hours

changed order of postprocessing so that the timeAverages are first and the seasonal timeSeries is last

can submit jobs to both irix and altix. Postprocessing will still be submitted to the AC's by default, as previous versions of frepp have done. If you want your job to submit postprocessing to the IC's instead, you will need to add an XML attribute to your postProcess tag as follows:

<postProcess platform="ia64">
...

Here are the conditions to consider when choosing a platform for your postprocesing:

added -m option from lol to allow creation of timeseries of runs less than one year which may or may not start on Jan. 1.

added maxyrs, a global variable telling the max chunkLength or interval to be calculated in this pp job

uses maxyrs to make the requested cpu time 1 hour if not creating any pp files containing more than one year of data

small changes and fixes to Analysis.pm in /home/fms/local/perl/v022/FRE: finds correct history directory when you specified it on the frepp command line, knows how to call rsh-acarch to make directories in /archive from linux, more organized stdout print statements

only test for the correct number of time levels in a file if the calendar is noleap

use nccatm rather than ncrcat for concatenating netcdf files

using the options -As (for figures only) will not submit a dummy postprocessing script.

Added support for segment length of 2, 3, and 4 months (in addition to 1, 6, and 12 which existed already.) frepp still does not perform postprocessing for runs with segments less than one month or greater than one year.

Preserves history files in $TMPDIR for the duration of a pp job

Additional dmgets to avoid krecalls

Prefer cpios to individual files when assembling timeseries from other timeseries (saves dmgets)

In creating a seasonal timeseries by concatenating shorter seasonal timeseries, check that each individual file exists before ncrcat, else send error message and skip the variable

Instead of doing seasonal ts at same interval as monthly ts and leaving individual seasons in archive until the seasonal chunk length has been reached, wait until least common multiple of monthly and seasonal chunks to do the seasonal ts, and only move chunked files to archive. (writes fewer intermediate files to archive)

changes to improve the metadata load on the system, buffer size for cpio increased from 64 to 256k, added "dmput -r" throughout

supports interpolating to the vertical levels in the atmospheric ERA40 data set. To get them, use

       <component type="atmos" zInterp="era40" source="atmos_month">

new zInterp option "zgrid" which calls Bob Hallberg's Resample_on_Z program. Usage is analogous to the atmostpheric resampling on pressure levels. Sample component:

       <component type="ocean_zgrid" source="ocean_month" zInterp="zgrid">
          <timeSeries freq="monthly" chunkLength="5yr">
             <variables> temp salt age u v </variables>
          </timeSeries>
          <timeSeries freq="monthly" chunkLength="20yr"/>
          <timeSeries freq="monthly" chunkLength="100yr"/>
          <timeSeries freq="annual" chunkLength="5yr">
             <variables> temp salt age u v </variables>
          </timeSeries>
          <timeSeries freq="annual" chunkLength="20yr"/>
          <timeSeries freq="annual" chunkLength="100yr"/>
          <timeSeries freq="seasonal" chunkLength="5yr">
             <variables> temp salt age u v </variables>
          </timeSeries>
          <timeSeries freq="seasonal" chunkLength="20yr"/>
          <timeSeries freq="seasonal" chunkLength="100yr"/>
          <timeAverage source="monthly" interval="5yr">
             <variables> temp salt age u v </variables>
          </timeAverage>
          <timeAverage source="monthly" interval="20yr"/>
          <timeAverage source="monthly" interval="100yr"/>
          <timeAverage source="annual" interval="1yr">
             <variables> temp salt age u v </variables>
          </timeAverage>
          <timeAverage source="annual" interval="5yr"/>
          <timeAverage source="annual" interval="20yr"/>
          <timeAverage source="annual" interval="100yr"/>
       </component>

If you do not explicitly state the variables as indicated above, then the following variables will be processed by default: temp,salt,age,u,v. Note that usually one doesn't need to specify

          <timeAverage source="annual" interval="1yr">

at all in the xml; it's just done automatically. And it is not required for ocean_zgrid either, however, if you leave it out frepp will give you a message that you didn't specify any variables for annual 1 year averages and it will use the default temp,salt,age,u,v.

now using Fanrong's Analysis.pm package for figures

modified not to cpio files when there is only one file

prevent trying to run platform csh intended for the altix on the sgi platform

added the ability to average daily data to a monthly timeseries. To take advantage of this capability, you need to add three lines to your xml:

         <component type="atmos" zInterp="ncep" start="0001" source="atmos_month">
            <timeSeries freq="daily" source="atmos_daily" chunkLength="5yr"/>
            <timeSeries freq="monthly" chunkLength="5yr"/>
            ...
            <timeSeries freq="monthly" averageOf="daily" chunkLength="5yr">
               <variables> t_ref_min t_ref_max </variables>
            </timeSeries>
         </component>

The new feature is 'timeSeries freq="monthly" averageOf="daily"'. There are several things to note:

gory bugfixy details surrounding making cpio files, calculating one or two variables offline after other variables have been processed, and performing frescrub at any point along the way.

the variables tag is now available for all timeSeries options.

checks that timeSeries contain the right number of time levels as it calculates them. If the timeSeries file contains the wrong number of levels, you'll get email about it.

ability to run on altix platform

analysis figure improvements

analysis figure bugfix when using <variables> tag inside of <timeSeries>; previously rtspp could mistakenly assume that pp files were missing and not create figures for that data

new timeSeries and timeAverage attribute 'from' can override what chunklength or interval pp you want to calulate a timeSeries or timeAverage from, ie

   <timeSeries freq="monthly" chunkLength="200yr" from="100yr"/>

annual multi-year averages will only try to calculate the interval they're supposed to calculate. Previously, they looked for the last existing output file and calculated all the intervals after that through the -t year. This change is only useful when you are trying to run offline pp out of order.

seasonal averages that need the previous december now know how to get it from the history file as well as from the previously existing monthly timeAverage file. This change is only useful when you are trying to run offline pp out of order.

better error message when you try to run seasonal timeSeries without any available monthly timeSeries data

-m option: If you give the -m option, you will get an email when your job is complete. Only useful for offline pp. Doesn't send email if option -p is used; this way you only get email when it's done with the stream

can put <variables> inside seasonal ts and it will use only those variables. Previously you could only use <variables> in the monthly ts, and those variables would then get used for seasonal and annual ts also. Note that you still need to have the monthly ts for all variables you request a seasonal ts of.

minor cleanup: seasonal ts tests whether cpio has already been extracted

BUGFIX: timavg.csh was using the wrong weighting interval in some cases

truncation of long filenames on timavg'ed files and plevel data avoided

bugfix for postprocessing jobs run after an experiment has been frescrubbed

dmget improvements to avoid the 'line too long' error

doesn't call nc_to_midtime, should improve speed of postprocessing by 10-25%

consolidates error notification into fewer actual emails

doublechecks that output directories exist so that there are no missing directories

additional dmgets implemented

static files created with splitncvars, should retain all appropriate attributes

time_bounds bugfix to seasonal averages when the first december of the run needs to be recalculated from the end of the first year

new analysis figure options:

          -Y year      = specify a four digit year as your analysis's start year, ex -Y 1984
                         This year overrides the startYear specifed in the <analysis> tag.
          -Z year      = specify a four digit year as your analysis's end year, ex -Z 1985
                         This year overrides the endYear specifed in the <analysis> tag.

analysis figures fixes

time_bounds fixes

bugfixes and improvements to external utilities:list_ncvars, timavg, plevel

developmental changes for time_bounds fixes

added option to split postprocessing into different jobs: -c component

ncrcat now uses -O option to overwrite files if they already exist. Previously if you were regenerating pp after a system error, you would get emails about ncrcat errors. ncrcat complained because the output file already existed. This change will prevent those error messages.

includes seasonal average functionality.

uses ncatted to put the calendar attribute back after plevel.sh called

filename attribute corrected in some annual average files

writes some version information to ppRootDir/.frepp_history