container.csh searched the whole list even after finding a match; I interrupted it after the first eureka.

But the split_hist function is calling container.csh too many times, should do it once and save it in a file.

Branch quebec_vb.

What happens when you run cp a b from several concurrent shells? Not sure. In any case, Amy has run into this problem of many concurrent hsmget calls on the same source. Essentially because of frepp calls from many components needing the same history file.

I have fixed it, or at least reduced it to the problem of many concurrent mv with the same destination, which hopefully linux handles correctly. The unpack-into-ptmp step is now done using a internal unique name and then renamed.

Branch quebec_vb.

Simon Marsland's code is in /home/sjm/ToyAusCOM/submodels/mom4p1/src/auscom_cpl.

The intent is to modify the solo driver mom4p1/drivers/ocean_solo.F90 only and run it as a separate executable managed by Prism.

• Is this Oasis3 or Oasis4?

All the changes are in mom4p1/drivers/ocean_solo.F90 and mom4p1/src/cpl_interfaces.F90:

vb 10:04am> cd /home/sjm/ToyAusCOM/submodels/mom4p1/src
vb 10:05am> grep -r -l -i prism .
./mom4p1/drivers/ocean_solo.F90
./auscom_cpl/cpl_interfaces.F90

Comments below refer to line number n:

Changes in mom4p1/drivers/ocean_solo.F90:

• Code is bounded by #ifdef AusCOM. Should this be #ifdef PRISM, anything AusCOM-specific?
• L181: prism_init before fms_init, should be ok.
• L194: for multi-exec, need independent namelist files.
• L226, also separate input directories?
• L421, unsure what to do with new ice routines, probably AusCOM-specific.

Changes in mom4p1/auscom_cpl/cpl_interfaces.F90:

• L7: should use use only for Prism modules.
• L83: replace print * with write( stdout() .... )
• L133: do not understand comment about "parallel coupling".
• subroutine prism_init looks ok, clean wrapper for Prism initialization.
• subroutine init_cpl looks ok, cleanly maps out domain2D data structure into Prism datatypes.
• L148: process log file writes from unit 85+my_task is dangerous! in FMS use getunit to get a free unit number.
• L263, comment says this is Oasis3. True?
• subroutine into_ice does Prism-based transfers from ocean to ice, appears to work single-threaded with global gather? (see mpp_global_field calls, L324). Probably inefficient. Maybe we need Oasis4 for parallel coupling? (I've been just given an Oasis4 paper to review, will see what I find...)
• into_ice also appears to have hardcoded coupling fields, Prism (or Oasis4) has developed external file-driven coupling spec.
• from_ice is the converse, also set to mono-process coupling via array vwork (this is hardcoded at L140).
• coupler_termination looks ok.

Overall comments:

• Looks good. Much thanks to the AusCOM folks.
• There isn't a clean separation between AusCOM and PRISM code, but it's clean enough that it should be distributed: user has to figure out how to customize things like from_ice and to_ice for their own needs.
• need to move to Oasis4 at some point.

Found out on the bbcp online manpage which is linked from NERSC's bbcp page that you can create a config file for commonly used options. You can use this set up a simple connection to p-09. Try putting this in a file called ~/.bbcp.cf:

-i /ccs/home/balaji/.ssh/id_dsa
-p
-S ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/bin/bbcp

on jaguar (will be shared by jaguarpf). The ssh identity file is created by typing ssh-keygen -t dsa. (On franklin you should add another line with -z).

Also put this in your .cshrc (or maybe we should put this in site/fre.cshrc)

setenv P09 vb@p-09.rdhpcs.noaa.gov:

Then the bbcp syntax simplifies to something like

bbcp -r $P09/staging-1/vb/CM2.4 .

These are modifications I've put in to compile on franklin, the pathscale compiler is particularly picky.

• modified mpp/include/mpp_error_s_a.h so that arguments are not ambiguous with mpp_error_a_s.h
• modified ocean_shared/generic_tracer/generic_tracer_utils.F90 and generic_topaz.F90 to make used a logical to match return code from send_data.
• modified ocean_shared/generic_tracer/generic_tracer_utils.F90 to convert allocated to _ALLOCATED, current code will not work unless _F2000 is set in fms_platform.h
• modified atmos_param/moist_processes/moist_processes.F90 so that namelist follows declarations, pathscale is picky about this.
• modified land_lad2/vegetation/vegn_data.F90 to prevent integer specifications of floating-point constants.
• land_lad2/river/river_physics.F90, River declared inout.

All the code changes are on branch perth_vb.

I am trying to find a quick path to running FRE on multiple platforms ("portable FRE"). The immediate need is to get us up and running — production, development, regression testing — on multiple platforms such as the HPCS and the DOE machines.

We foresaw the need for a platform tag but so far have not used it for much. I've revived this tag within the setup node: for instance, see /home/vb/fre/xml/CM2.4C_static_v2.xml,

    <platform name="pathscale">
      <directory type="root">$HOME/fre/perth/CM2.4</directory>
      <directory type="site">$HOME/fre/site</directory>
      <mkmfTemplate file="$HOME/fre/site/mkmf.template.x86_64" />
      <csh>
        source $MODULESHOME/init/csh
        module switch PrgEnv-pgi PrgEnv-pathscale
        module load netcdf-pathscale
        module list
      </csh>
    </platform>
    <platform name="pgi">
      <directory type="root">$HOME/fre/perth/CM2.4</directory>
      <directory type="site">$HOME/fre/site</directory>
      <mkmfTemplate file="$HOME/fre/site/mkmf.template.pgi" />
      <csh>
        source $MODULESHOME/init/csh
        module load netcdf
        module list
      </csh>
    </platform>
    <platform name="hpcs">
      <directory type="root">$HOME/fre/perth/CM2.4</directory>
      <directory type="archive">/archive/$USER/fre/perth/CM2.4</directory>
      <directory type="site">$HOME/fre/site</directory>
      <directory type="work">$TMPDIR</directory>
      <directory type="analysis">/net2/$USER/fre/perth/CM2.4</directory>
<!--       <project>gscip</project> -->
      <mkmfTemplate file="$HOME/fre/site/mkmf.template.ia64_flt" />
      <csh>
        source /opt/modules/default/init/tcsh
        module purge
        module load modules
        module load ifort.9.1.051
        module load icc.9.1.045
        module load scsl-1.5.1.0
        module load idb.9.0.12
        module load totalview_730-0
        module load mpt-1.14
        module load netcdf-3.6.2
      </csh>
      <getFmsData>/home/bw/fms/scripts/get_fms_data_hadisst</getFmsData>
      <fmsRelease>perth</fmsRelease>
    </platform>

• Notice that platform now allows you to set script defaults on a per-platform basis, including picking the mkmfTemplate. The code in FRE.pm is now written so that if it doesn't find a platform node under setup, it uses setup itself. This code construct could be used to add new tags fairly easily:

# read the setup node
   my $setup = $root{$fre}->findnodes('setup')->get_node(1);
# read the platform node into $base
   my $base = $setup->findnodes("platform[\@name=\"$platform\"]")->get_node(1)
     || $setup;                 # this will provide backward-compatibility

• The platform is understood to mean the complete hardware/software environment. (pathscale and pgi pick two different compiler options available on the Cray machines at DOE: we don't know yet which is better).
• I've modified fremake and frerun to use a new command-line option to pick which platform to run on (no default). With this, the same FRE input file will work on any platform. One question about the CuratorDB: it needs to record which platform you actually used: how do we record this? Does the "canonical XML" FRE file only record the one platform tag that was used?
• Site-specific command syntax (e.g qsub for interacting with the scheduler; mpirun vs. aprun for launching a parallel job, is encoded in the site config file (currently site/fre.cshrc).
• To make frerun fully functional on the DOE machines, we need the access to input files to be platform-dependent. We have also talked about "versioning" of input files. I'm wondering if it's possible to do a quick implementation of the dataFile node from FRE4?

<dataFile target="INPUT/foo" source="/full/path/foo" checkSum timeStamp>

• Should this line be enclosed in a platform tag? or should it have a platform attribute?
• do we need the label attribute, or is target sufficient?
• I'm hoping it will be possible to implement this new feature without losing backward compatibility, as done for the setup/platform tags.