11.3.4 Questions and Answers

Here are some questions and answers about memory window and related issues:

• How much space does the memory window require?

  There are 12 arrays required to hold three time levels of prognostic data for two horizontal velocity components and two tracers within the memory window. The arrays are dimensioned as A(imt,km,jmw). For the simplest options, the workspace part of the memory window must allow space for 3 advective velocities on the faces of T-cells, 3 advective velocities on the faces of U-cells, 3 temporary arrays for use as fluxes on cell faces, 1 array for density, and 2 arrays for land/sea masks. If it is assumed that all of these 12 arrays in the workspace part are dimensioned as above, then 50% of the space within the memory window is workspace and 50 assumption when the memory window is fully opened (jmw=jmt for second order numerics). The percent of space taken by workspace arrays is significantly less when the window is minimum (jmw=3). But the minimum window is not the constraining configuration.

• What are the basic ways to use the memory window and how do the space requirements compare?

  Refer to Fig 11.8a which is a typical situation when MOM 3 is configured to solve a problem which is too big to fit into memory on a conventional vector platform such as the CRAY T90 at GFDL. All latitude rows are placed on disk. This is viable only if the disk is a solid state device such as SSD on the CRAY^11.10. The option for keeping data on solid state disk on the CRAY T90 is ssread_sswrite.

  Figure 11.8b indicates what happens when option ramdrive is used instead of option ssread_sswrite. This option is appropriate for platforms that do not have solid state disk. Note that in comparison to figure 11.8a, the memory has increased dramatically and no disk space is required. The ramdrive is just an array dimensioned large enough to hold all latitude rows. Reads and writes to the ramdrive are replaced by memory to memory copies.

  Figure 11.8c is the situation when the memory window is fully opened up. The options for this configuration is radmrive and max_window. In this case, all latitude rows are stored directly in the memory window. For large models with many latitude rows, the fully opened memory window takes three times as much memory space as the case in figure 11.8b. For more detail, refer to Section 37.1.

• When the memory window is opened all the way, is the global index ``jrow'' the same as the local window index ``j''?

  Yes and no. When fully opened, the number of latitude rows in the memory window is given by jmw=(jmt-2) + 2*jbuf. Consider the case of one processor: global arrays are dimensioned as A(imt,1:jmt) and memory window arrays are dimensioned as B(imt,km,1:jmw). For a second order window (when equations require second order numerics), jbuf=1 so jmw=jmt and jrow=1 references the same latitude row as j=1 in the memory window array.

  However, the same is not true for a fourth order window. The reason is that jbuf=2 which makes the memory window arrays dimensioned as B(imt,km,1:jmt+2) while global arrays are still dimensioned as A(imt,1:jmt). The relationship between indices is such that jrow=1 in the global arrays references the same latitude row as j=2 in the memory window arrays.

  To make the local and global latitude indices the same for all cases, the memory window arrays would have to be made allocatable and dimensioned as B(imt,km,jscomp-2*jbuf:jecomp+2*jbuf) where ``jscomp'' and ``jecomp'' are a function of processor. Allocating these arrays as such currently slows the model down by 30%.

• Can space be saved by dimensioning workspace arrays as B(imt,km) instead of B(imt,km,jmw)?

  Yes. All ``j-loops'' could be replaced by one large ``j-loop'' wrapped around all routines needed to compute internal modes and tracers. However, there are problems with this technique when message passing on multiple processors. Essentially, the computation becomes ``serialized'' when processor ``n'' must wait for adjacent processor ``n-1'' to finish before inter-processor communication can be used to extract correct quantities at domain boundaries. This happens, for instance, when quantities are computed incorrectly at domain boundaries because the proper inputs are unavailable. The standard solution is to replace incorrectly computed quantities with correct ones from adjacent domains via inter-processor communication. Alternatively, the memory window can be bumped up to the next higher order to include more buffer rows. Another problem with using the single ``j-loop'' technique is the arrangement is not flexible enough to cover all possibilities. For instance, option pressure_gradient_average which requires tracers to be solved to the north of velocity cells before baroclinic velocities are solved is problematic.

• Can moving of data within the memory window be eliminated by using indices and rotating the indices as the window is moved northward?

  Yes. If a new set of indices ``jm1,jc,jp1'' are introduced to refer to rows ``j-1,j,j+1'' in the memory window, then instead of moving data, the new indices can be rotated. This technique would also save memory. There are two disadvantages: for higher order schemes, new indices need to be introduced (i.e. ``jm2'' and ``jp2'' for forth order schemes); and the scheme is prone to errors. For instance, if an array is referenced with ``j+1'' instead of ``jp1'', then errors result.

figure=slice.ps,height=8.0in

 

figure=mw2.ps,height=8.0in

  

Figure 11.2: a) Detailed and simplified schematic of the minimum sized memory window with jmw=3 latitude rows for 2nd order numerics. b) Memory window opened up to hold jmw=6 latitude rows.

figure=mw34.ps,height=7.5in

  

Figure 11.3: a) Minimum sized memory window for 4th order numerics.

figure=otsmw2a.ps,height=8.0in

  

Figure 11.4: a) Loading the memory window, solving prognostic equations on the central row and writing $\tau+1$ data back to $\tau-1$ disk for latitude row jrow=4. b) Solving prognostic equations for all latitude rows from ``jscomp'' to ``jecomp'' by moving the memory window northward.

figure=otsmw4a.ps,height=8.0in

  

Figure 11.5: a) Loading the memory window, updating the central row and writing to disk for latitude row jrow=4. b) Updating all latitude rows by moving the memory window northward.

figure=mwcopy.ps,height=7.0in

  

Figure 11.6: The memory window is moved northward by copying data from the northernmost rows into the southernmost rows followed by reading of new data. a) Various types of rows within the memory window. b) Copy operation for a 2nd order memory window with jmw=3. c) Copy operation for a 4th order memory window with jmw=5. d) Copy operation for a larger 2nd order memory window. e) Copy operation for a larger 4th order memory window.

figure=mwcopy2.ps,height=7.0in

  

Figure 11.7: The memory window is moved northward by copying data from the northernmost rows into the southernmost rows followed by reading of new data. Note that only rows where baroclinic velocities are computed are written back to disk (or ramdrive). a) Various types of rows within the memory window. b) Copy operation for the first three memory windows of minimum size jmw=6 when option pressure_gradient_average is enabled. c) Copy operation for the first three memory windows of a larger size jmw=8 when option pressure_gradient_average is enabled.

figure=configs.ps,height=7.0in

  

Figure 11.8: Various ways of configuring MOM illustrating the space used on disk and in memory. a) Options fio and ssread_sswrite. b) Option ramdrive. c) Options ramdrive and max_window.