Contact:  V. Balaji
Reviewers: 
Change History: WebCVS Log
RCS Log: RCS Log

Scalable implementations of finite-difference codes are generally based on decomposing the model domain into subdomains that are distributed among processors. These domains will then be obliged to exchange data at their boundaries if data dependencies are merely nearest-neighbour, or may need to acquire information from the global domain if there are extended data dependencies, as in the spectral transform. The domain decomposition is a key operation in the development of parallel codes.

mpp_domains_mod provides a domain decomposition and domain update API for rectilinear grids, built on top of the mpp_mod API for message passing. Features of mpp_domains_mod include:

Simple, minimal API, with free access to underlying API for more complicated stuff.

Design toward typical use in climate/weather CFD codes.

Domains

I have assumed that domain decomposition will mainly be in 2 horizontal dimensions, which will in general be the two fastest-varying indices. There is a separate implementation of 1D decomposition on the fastest-varying index, and 1D decomposition on the second index, treated as a special case of 2D decomposition, is also possible. We define domain as the grid associated with a task. We define the compute domain as the set of gridpoints that are computed by a task, and the data domain as the set of points that are required by the task for the calculation. There can in general be more than 1 task per PE, though often the number of domains is the same as the processor count. We define the global domain as the global computational domain of the entire model (i.e, the same as the computational domain if run on a single processor). 2D domains are defined using a derived type domain2D, constructed as follows (see comments in code for more details):

     type, public :: domain_axis_spec
        private
        integer :: begin, end, size, max_size
        logical :: is_global
     end type domain_axis_spec
     type, public :: domain1D
        private
        type(domain_axis_spec) :: compute, data, global, active
        logical :: mustputb, mustgetb, mustputf, mustgetf, folded
        type(domain1D), pointer, dimension(:) :: list
        integer :: pe              !PE to which this domain is assigned
        integer :: pos
     end type domain1D
domaintypes of higher rank can be constructed from type domain1D
typically we only need 1 and 2D, but could need higher (e.g 3D LES)
some elements are repeated below if they are needed once per domain
     type, public :: domain2D
        private
        type(domain1D) :: x
        type(domain1D) :: y
        type(domain2D), pointer, dimension(:) :: list
        integer :: pe              !PE to which this domain is assigned
        integer :: pos
     end type domain2D
     type(domain1D), public :: NULL_DOMAIN1D
     type(domain2D), public :: NULL_DOMAIN2D

The domain2D type contains all the necessary information to define the global, compute and data domains of each task, as well as the PE associated with the task. The PEs from which remote data may be acquired to update the data domain are also contained in a linked list of neighbours.

     mpp_parameter_mod
      mpp_datatype_mod
          mpp_data_mod
  mpp_domains_util_mod
mpp_domains_reduce_mod
  mpp_domains_misc_mod
mpp_domains_define_mod

None.

None.

None.

None.

Any module or program unit using mpp_domains_mod must contain the line

     use mpp_domains_mod

mpp_domains_mod uses mpp_mod, and therefore is subject to the compiling and linking requirements of that module.

mpp_domains_mod uses standard f90, and has no special requirements. There are some OS-dependent pre-processor directives that you might need to modify on non-SGI/Cray systems and compilers. The portability of mpp_mod obviously is a constraint, since this module is built on top of it. Contact me, Balaji, SGI/GFDL, with questions.

The source consists of the main source file and also requires the following include files: GFDL users can check it out of the main CVS repository as part of the CVS module. The current public tag is . External users can download the latest package . Public access to the GFDL CVS repository will soon be made available.

None.

None.

None.

None.

top