Users Guide

From Foam
Revision as of 10:24, 2 January 2007 by Mickelso (talk | contribs)
Jump to navigationJump to search

FOAM Users Guide

This Guide began as the "Notes on using FOAM". Users are encouraged to add to this guide to fill in needed details.

Quick Start

Many coupled systems come with one massive shell script which combines the build system and run system: it compiles the code, copies data files and executables into place, and runs the model. FOAM keeps the compile and run systems separate. This makes development somewhat easier because the build system consists of just makefiles and source code. For runtime, FOAM does not provide a one-script-does-all shell script because the parallel platforms FOAM runs on offer so many different choices for queue software, filesystems, mass store, etc. that an all purpose script, if it was possible to build, would be excessively long and hard to debug.

Instead, we describe below the general procedure for setting up a FOAM run directory. If you choose, you can incorporate this into a shell script appropriate for your system.

1. Choose a directory to run the model. We'll call it "foamrun". It should have plenty of disk space for the output.

2. Obtain the boundary/initial condition file:

  • foam1.5datR15_128.single.be.tar (Big Endian. For IBM, SGI, Sun)
  • foam1.5datR15_128.single.le.tar (Little Endian. For Linux)

(foam1.5dat*) from the FOAM ftp site and untar in this directory. A directory called "f1.5data" will be created.

3. Use "mkdir" to create the following subdirectories to hold the history and restart output. (FOAM expects this directory structure to exist)

  • history
    • history/atmos
    • history/coupl
    • history/ocean
  • restart
    • restart/atmos
    • restart/coupl
    • restart/ocean

4. Place the restart data set from the ftp site:

  • rs.F1_5.modern.be.1440.tar (Big Endian. For IBM, SGI, Sun)
  • rs.F1_5.modern.le.1440.tar (Little Endian. For Linux)

in the restart subdirectory and untar. There is a separate file for each model (two for the atmosphere). The coupler restart contains restart data for the land, sea-ice, and river models.

(NOTE: this example assumes you are doing a restart run. See below for an initial run.)

5. Place the files run_params, atmos_params (samples available at the ftp site: atmos_params, run_params) and the foam executable (foam1.5) in foamrun.

6. Edit the file run_params and change the default PREFIX with the complete path to your foamrun directory. replace TIME_INV with the complete path to the data directory created above.

7. Check the value of FINISHED in run_params. It should be the same number as at the end of the restart files.

8. Edit HISTFRQ and RESTFRQ to adjust the history and restart frequency to what you want.

You are now ready to run. See below for information on how to start the model.

NOTE: Its possible to locate "foamrun", the "f1.5data" directory and the history and restart directories each on different file systems. For "foamrun" and "f1.5data", just enter the complete path in run_params. See below for moving the history and restart directories.

Summary of FOAM run directory contents:

Before starting a run, the run directory contents should be:

  • atmos_params
  • run_params
  • foam1.5*
  • history/
    • history/atmos/
    • history/coupl/
    • history/ocean/
  • restart/
    • restart/atmos/restart.atmos.xxxxxx
    • restart/atmos/restart.atmos.xxxxxx.A
    • restart/ocean/restart.ocean.xxxxxx
    • restart/coupl/restart.coupl.xxxxxx

run_params contents:

  • PREFIX: path for the run directory
  • STORAGE: Currently ignored. Use same path as run directory.
  • TIME_INV: path to the data directory.
  • RESTFRQ: Frequency of restart writes in days.
  • HISTFRQ: Frequency of history writes in days.
  • FINISHED: The day postfix of the restart file to start from
  • RUNLNG: The number of days to run.
  • INITIAL: T or F. Indicate initial run
  • FILTPHIS: T or F. Filter topography. Use only for initial run.

Note: RUNLNG should be a multiple of RESTFRQ but the model doesn't check for this. Also RESTFRQ should be a multiple of HISTFRQ.

For a 5 year restart run with monthly histories and annual restarts, run_params would look like:

  • RESTFRQ: 360
  • HISTFRQ: 30
  • PREFIX: dir
  • STORAGE: dir
  • TIME_INV: dir
  • FINISHED: 0001440
  • RUNLNG: 1800
  • INITIAL: F
  • FILTPHIS: F

(NOTE: 1800 = 5*360. FOAM uses a 360-day year.)

Moving the history/restart directories

You may have the history and restart data in a different directory (or filesystem) than the run directory. Suppose you create a directory called storedir. Create "history" and "restart" subdirectories and the atmos,coupl,ocean subdirectories as indicated above. Move your restart files to the new location. Use the UNIX command "ln" to set a link from the run directory to the new location of "history" and "restart". Example: While in the run directory, type: "ln -s <storedir>/restart" and "ln -s <storedir>/history"

Running the Model

Start the model with a command like (assuming you're using the 32 proc configuration):

mpirun -np 32 foam1.5 run_params

run_params is an argument that foam expects.

Other versions:

mpirun -np 32 foam1.5 run_params >& std_out. The redirection (>&) sends stdout and stderr from mpirun to a file called std_out.


If you are using a queue system you will need to create a "run script". Consult the local documentation. A line like the above will need to be in the script to start the model.

When Finished

When the run finishes, there will be several files in the run directory:

  • Finished -- A 0 size file written to indicate model has finished
  • Restart.xxxxx -- 0 size files written to indicate when a restart set is written
  • pccm.error.xxxx -- Atmosphere model stderr files. One for each processor
  • pccm.out.xxxxxx -- Atmosphere model stdout file.
  • om3.err.xxxx -- Ocean model stderr. One for each processor (In colocated mode, these files are not opened)
  • om3.out.xxxx -- Ocean model stdout
  • std_out -- the redirected mpirun std_out

There will also be soft links to the atmosphere restart files (links are called rest and rest.A)

You should delete all of the above when preparing the run directory for a new run.

There should be history.model.xxx files in the history subdirectories and new restart files in the restart directories. To continue the run, change FINISHED in run_params to the day number of the last-written restart set. If you started with an initial run, change INITIAL and FILTPHIS to F. Then reissue the mpirun command or resbumit the run script.

Pccm.out output

The pccm.out.* file containts statistics from the atmosphere written every timestep.

example: NSTEP = 4872961 8.889008313417435E-05 7.462289431714453E-06 2.521384124755859E+02 9.847287500000000E+04 2.544476318359375E+01 5.455936193466187E-01

what each number means:

  • NSTEP = 4872961 the total number of timesteps integrated
  • 8.889008313417435E-05 Global RMS Vorticity
  • 7.462289431714453E-06 Global RMS Divergence
  • 2.521384124755859E+02 Global RMS Temperature (Kelvin)
  • 9.847287500000000E+04 Global Mass Integral
  • 2.544476318359375E+01 Global Moisture Integral
  • 5.455936193466187E-01 Maximum Courant number

Initial Run

By default, FOAM is always in restart mode and will try to read a set of restart files when starting. If you want to change the boundary conditions, you will need to do an initial run. See below for how to change the boundary conditions. Here's how to do an initial run:

1. edit run_params and add or change the following line: INITIAL: T

2. If your new topography file is from Slarti or from another source and it has not been spectrally filterd, add the following line to run_params FILTPHIS: T

Note: If you are using the original NCAR-supplied initial files, those have PHIS already filtered.

INITIAL: T forces all models to do an initial run. It will override the value of NSREST in atmos_params. NSREST is now ignored.

Om3 initial state: Om3 will initialize its temperature and salinity from om3.salt and om3.temp in the f1.5data directory. Velocities will be 0 everywhere.

slab ocean initial state: The slab will initialize its temperature from the file sstinit.ascii in the f1.5data directory. Velocities will be 0 everywhere.

Atmosphere initial state: The atmosphere will try to read initial data from the file NCDATA in atmos_params. Typically this is SEP1* in the f1.5data directory. NOTE: If you are using the original NCAR-supplied SEP1 files for initial data, you will need to set -DFORCEHY in FOAM_OPTS in the Makefile to force the correct vertical level values. These values are correct in Slarti output.

Land/Sea Ice/River initial state: The initial state for the land, sea ice and river models are set within FOAM and not read from a file. Land temperature has a zonally uniform cosine profile that is constant with depth. Soil moisture and snow depth are set to 0. Sea ice is absent everywhere. Rivers are empty.

At the end of the initial run, each model should output restart files which can then be used in subsequent runs.

see continue the run above.

Compiling on Different Platforms

The makefile is set for compiling a 32 processor executable, single precision, with an R15 atmosphere and 128x128 ocean/land surface.

You will need GNUmake or a compatible make version. GNUmake is sometimes called "gmake" on non-Linux systems. The source code for GNUmake is available here.

The Makefile uses "uname" to discover what platform it is on and set appropriate compiler settings. IBM, SGI, Linux and Sun are recognized. To add a new platform, add a section to the top level Makefile similar to those for the supported platforms.

Linux platforms often have many different compilers. FOAM's Makefile assmes the Portland Group Fortran compiler is used. To switch to the Intel compiler, you will have to edit the Makefile and comment/uncomment the appropriate lines for the compiler flags.

Compiling and Running with NetCDF

In the Makefile, add -DNETCDF to FOAM_OPTS and uncomment the three lines for the NetCDF libraries and include paths. You will need to edit the two paths below FOAM_OPTS

  • NETCDF_INC= -I/soft/apps/packages/netcdf-3.6.0-p1-pgi-5.2/include
  • NETCDF_LIB= -L/soft/apps/packages/netcdf-3.6.0-p1-pgi-5.2/lib

to point to your local installation of NetCDF. Do a "make clean" and "make". This will activate both the NetCDF input and output.

In your run directory, edit atmos_params and change the 4 input files to NetCDF versions:

  • NCDATA = 'SEP1.R15.nc',
  • BNDTI = 'tibds.R15.nc',
  • BNDTVS = 'tvbds.R15.nc',
  • BNDTVO = 'ozn.R15.nc',

Note that BNDTVS is not needed in FOAM; only in the standalone atmosphere. NCDATA is only read during an initial run.

You can make NetCDF versions of the atmosphere data by running "ccm2nc" on the binary input files and leaving them in your f1.5data directory.

You will also need the NetCDF versions of the land/river boundary data. These can be made with the program CplBC2nc.tar.gz

NetCDF versions are also available from the ftp site (see Boundary/Initial Condition data below.)

You can switch to the NetCDF input version and maintain bit-for-bit identical results with the binary input.

Changing the Processor Decomposition

Below is an example list of allowed decompositions for the standard configuration of R15 and 128x128. You'll need to edit the FOAM_OPTS string in the Makefile accordingly and recompile for the changes to take effect.

The nuber of processors in each direction (lat and lon) must always be a power-of-two.

Note that the land, sea-ice, river model and coupler always run on the same processors as the atmosphere. Non-Colocated options (ocean and atmosphere-coupler-land-ice on two distinct sets of processors) Note: you can add more processors to the ocean for each non-colocated case below but each number (lat and lon) must be a power-of-two. There are no other options for the atmosphere processor count (for R15) other than what is indicated.

2x2 + 1x1 = 5 total
-DOCEAN_LAT_PROCS=1\
-DOCEAN_LON_PROCS=1\
-DATMOS_LAT_PROCS=2\
-DATMOS_LON_PROCS=2\

2x4 + 1x1 = 9 total
-DOCEAN_LAT_PROCS=1\
-DOCEAN_LON_PROCS=1\
-DATMOS_LAT_PROCS=2\
-DATMOS_LON_PROCS=4\

4x4 + 1x1 = 17 total
-DOCEAN_LAT_PROCS=1\
-DOCEAN_LON_PROCS=1\
-DATMOS_LAT_PROCS=4\
-DATMOS_LON_PROCS=4\

4x8 + 1x2 = 34 total
-DOCEAN_LAT_PROCS=1\
-DOCEAN_LON_PROCS=2\
-DATMOS_LAT_PROCS=4\
-DATMOS_LON_PROCS=8\

4x16 + 2x2 = 68 total
-DOCEAN_LAT_PROCS=2\
-DOCEAN_LON_PROCS=2\
-DATMOS_LAT_PROCS=4\
-DATMOS_LON_PROCS=16\

Co-located option (ocean and atmosphere located on the same processors)

Note: Co-located requires that (OCEAN_LAT_PROCS * OCEAN_LON_PROCS) = (ATMOS_LAT_PROCS * ATMOS_LON_PROCS) and that you also set the cpp constant -DCOLOCATED_OCEAN. Note that you can use some other combination of ocean processors other than what is indicated as long as the rule above still holds.

2x2 = 4 total
-DOCEAN_LAT_PROCS=2\
-DOCEAN_LON_PROCS=2\
-DATMOS_LAT_PROCS=2\
-DATMOS_LON_PROCS=2\
-DCOLOCATED_OCEAN\

2x4 = 8 total
-DOCEAN_LAT_PROCS=2\
-DOCEAN_LON_PROCS=4\
-DATMOS_LAT_PROCS=2\
-DATMOS_LON_PROCS=4\
-DCOLOCATED_OCEAN\

4x4 = 16 total
-DOCEAN_LAT_PROCS=4\
-DOCEAN_LON_PROCS=4\
-DATMOS_LAT_PROCS=4\
-DATMOS_LON_PROCS=4\
-DCOLCOATED_OCEAN\

4x8 = 32 total (DEFAULT)
-DOCEAN_LAT_PROCS=4\
-DOCEAN_LON_PROCS=8\
-DATMOS_LAT_PROCS=4\
-DATMOS_LON_PROCS=8\
-DCOLCOATED_OCEAN\

4x16 = 64 total
-DOCEAN_LAT_PROCS=8\
-DOCEAN_LON_PROCS=8\
-DATMOS_LAT_PROCS=4\
-DATMOS_LON_PROCS=16\
-DCOLOCATED_OCEAN\

Compile Time Options for Model Configuration/Physics

See the top level Makefile for an explanation of model configuration options. Type "make help" to see what targets are available.

How to run a fixed-sst configuration Fixed-sst runs are accomplished using a version of the slab model.

1. Edit the top-level Makefile and add the following -DSST_FROM_FILE -DSIMSICE to the FOAM_OPTS variable.

2. type "make clean; make foamslab"

When the build is finished you will have an executable called "foam1.5.slab". With the options above, this version of the model will force the atmosphere with SST's from the file "sstshea5c" which is found in the f1.5data directory. Sea-ice extent is determined by the SST field. Sea ice is placed whereever the SST is below freezing. The simple sea ice model is used to predict temperatures in the sea ice. "sstshea5c" contains 12 monthly averages of observed SST's. The slab ocean will interpolate between between each month to provide SST values at each ocean time step. Land and river models operate exactly as in fully-coupled mode.

Runtime Options for Model Configuration/Physics

There are currently no runtime options for changing the model configuration.

Changing atmospheric constants

Gas concentrations , the solar constant and aerosal optical depth can now be changed at runtime through the atmosphere model's namelist, atmos_params. Since these values are not saved in the atmosphere restart file, they will take effect in both restart and inital runs. Other runtime atmosphere parameters, such as timestep and diffusion, can only be changed with a new inital atmosphere run.

Example of what you would change or add in atmos_params to change all gas constants to Pre-industrial conditions.

CO2VMR = 2.80e-4,
N2OVMR = 0.275e-6,
CH4VMR = 0.7e-6,
F11VMR = 0.,
F12VMR = 0.,

Example of 1975 conditions.

CO2VMR = 3.35e-4,
N2OVMR = 0.302e-6,
CH4VMR = 1.500e-6,
F11VMR = 0.13e-9,
F12VMR = 0.30e-9,

An example of changing the solar constant and aerosal optical depth. SCON = 1.360e6,
TAUVIS = 0.10,

NOTE TAUVIS is not really the aerosol optical depth. Read this note before adjusting this parameter.

An example of setting the orbital parameters (and what they mean): ECCEN = .018682, (Eccentricity of orbit)
OBLIQ = 24.1054, (Earth's obliquity angle)
MVELP = 179.1304, (Degree's between vernal equinox and perihelion)
DAYP = 259.003, (number of day from jan1(modern) to date of perihelion)

NOTE: The model will use default values for each of these variables not present in atmos_params. Default values are: CO2VMR = 3.55e-4,
N2OVMR = 0.311e-6,
CH4VMR = 1.714e-6,
F11VMR = 0.280e-9,
F12VMR = 0.503e-9,
SCON = 1.367e6,
TAUVIS = 0.14,
ECCEN = .016724,
OBLIQ = 23.4463,
MVELP = 77.9610,
DAYP = 2.0


Changing atmospheric history file number and contents In PCCM3 terminology, the main history file output by the atmosphere, history.atmos.*, is the "primary" history file. You can have up to 5 auxiliary files with different content and frequencies (I've only tried 2 auxiliary files).

To turn on auxiliary files, you need to edit atmos_params. In this example, we'll add 1 auxiliary file for daily PSL and SHFLX with 30 writes per file.

First, in atmos_params, change:

MFILT =1,
to
MFILT = 1,30,

MFILT sets the number of history writes per file. The first number is for the primary history file and the second is for the first auxiliary file.

Now add the following:

NHTFRQ = -24, -24,
AUXF = '1','PSL','SHFLX',

The integers in NHTFRQ set the history write frequencies. The first number is for the primary file and is ignored in favor of HISTFRQ in run_params. The second number sets the auxiliary file frequency at once per 24 hours (negative = hours, positive = timesteps). You could also set NHTFRQ as:

NHTFRQ = ,-24,

Finally, AUXF = '1','PSL','SHFLX' sets the content of the auxiliary file to PSL and SHFLX.

The output files in history/atmos will be called ha0001, ha0002, etc. You can convert these to netcdf with ccm2nc.

Ex: For two auxiliary files with PSL daily and every timestep and one file each per year:

MFILT = 1,360,17280
NHTFRQ = -24,-24,1
AUXF = '1','PSL','2','PSL'

Output will be ha0001, hb0001, ha0002, hb0002, etc.

A summary of the AUXF settings will also be reported in pccm.out.*

The contents of the primary history file are initially set in the code. You can alter those contents at runtime with the PRIMARY and EXCLUDE variables in atmos_params. PRIMARY will *add* fields to the history file wile EXCLUDE will remove them.

Example. To remove omega and add relative humidity and the longitude pressure gradient to the primary atmosphere history, add the following lines to atmos_params: PRIMARY = 'RELHUM','DPSLON',
EXCLUDE = 'OMEGA',


Boundary Data

Retrieved from "http:/// /FOAM/index.php?title=Users_Guide&oldid=1376"