==Introduction==

We currently provide two Linux kernels because of GPFS support on ION.

* 2.6.19 based kernel : Zepto CN kernel
** IBM V1R3 patch and zepto patch applied
** 64KB pagesize and bigmemory region available
** Device drivers for compute node devices such as DMA, lockbox, etc
** Allow to run MPICH/DCMF code through Zepto Compute Binary(ZCB)
** Can be used as enhanced ION kernel (no GPFS modules available)

* 2.6.16 based kernel : Zepto ION kernel
** IBM V1R3 patch applied
** Essentially same as IBM ION kernel. Support GPFS

==Kernel directory structure==

Kernel directory basically consists of three main directories ; prebuilt, config and tarball.

<pre>
kernel
|-- prebuilt
| |-- 2.6.16
| | `-- ION
| `-- 2.6.19
| |-- CN
| `-- objs
|-- tarball
`-- config
</pre>

The prebuilt directory contains prebuilt kernel images and kernel modules. While prebuilt ION kernel
ELF file is found, no CN kernel ELF file is found here. This is because CN kernel ramdisk is embedded in kernel
and we wanted to provide a way to replace the CNK ramdisk without invoking CN build process which requires
untar'ing source code. Instead CN kernel objects are found in prebuilt directory and
a CN kernel ELF is created from the objects and ramdisk every time when you invoke certain make target.

The tarball directory contains kernel tarballs separately for ION and CN Linux kernel. Technically, those tarball are a snopshot of
the Zepto kernel git repository. The directory might contain a .patch file that contains difference between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git even for small modification.
Associated git log file can be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot is created, so this information is used to create a patch file.
<pre>
linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2
</pre>

The config directory contains Linux kernel config. A config file is basically associated to a tarball by kernel version.
i.e, bgp-cn-2.6.19.2-dot-config for linux-2.6.19.2-BGP-V1R3.tar.bz2
See also Makefile.

==Build kernel==

Makefile in the kernel directory has many options. Just type 'make' it will show you help message.

If you need to build (or rebuild) kernel from the tarball of kernel source code, use '''bgp-ion-linux-build''' or '''bgp-cn-linux-build target'''.
By default it extracts ION or CN kernel tarball in a directory named work, apply a patch if any
and start kernel build. Once kernel has successfully been built, kernel images
(in both Zepto top directory and tmp directory) will be replaced with newly built images.
ION kernel source code is extracted in '''work/linux-2.6.16.46-297-BGP-V1R3''' and
CN kernel source is in '''work/linux-2.6.19.2-BGP-V1R3'''.
Here is an example of build, rebuilding kernel.

<pre>
$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
</pre>

===Build kernel from Zepto kernel git repo===
As mentioned above the kernel tarball is used as source code by default.
If you specify '''GIT=1''' to the make command,
you can directly build from our Zepto kernel git tree. This is very useful for kernel development since
you can keep track your modifications.

<pre>
$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....
</pre>

Note that repo/linux-2.6.19.2-BGP-V1R3 is a cloned git repository that is cloned from
http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/.
Our http repo is read-only, so you can't push your modifications to the http repo.
You can post your git patch to [mailto:zeptoos@mcs.anl.gov ZeptoOS mailing list] first.

See also [http://bg-linux.anl-external.org/wiki/index.php/Main_Page BG-Linux page] for our kernel git repo details.

===Kernel config===

Initially, config/bgp-cn-2.6.19.2-dot-config is applied CN Linux kernel build tree
and config/bgp-ion-2.6.16.46-dot-config is applied to ION Linux kernel build tree.
Technically when you type a kernel build target the first time, associated kernel config file is copied to .config in
kernel build directory.

Here is the location of kernel config file.
* Regular build
** work/build-2.6.19.2-BGP-V1R3/.config
** work/build-2.6.16.46-297-BGP-V1R3/.config
* GIT build
** repo/build-2.6.19.2-BGP-V1R3/.config
** repo/build-2.6.16.46-297-BGP-V1R3/.config

Please note that the kernel config file is copied only once until you do distclean or
remove the files manually.

bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig target invoke text based Linux kernel configuration menu.

<pre>
$ make bgp-ion-linux-menuconfig
$ make bgp-cn-linux-menuconfig
</pre>

For GIT build,
<pre>
$ make GIT=1 bgp-ion-linux-menuconfig
$ make GIT=1 bgp-cn-linux-menuconfig
</pre>

Those menu target never update default kernel config file in the config directory.
If you want to apply new config permanently, please copy it to the config directory by hand.
For example,
<pre>
$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config
</pre>

==Kernel (command line) Parameters==

In usual server/desktop Linux environment, kernel parameters
are passed via bootloader such as grub. However, BlueGene/P boot mechanism
does not have such capability. Kernel parameters are very handy since we can skip kernel rebuild.
So we have modified CN Linux kernel (2.6.19) to allow to load kernel parameter string
embedded in kernel ELF image file.

You can add kernel parameters (or reset) to a kernel ELF file by
a command line tool named zkparam.py, which is located in kernel directory.
Here is the synopsis of the tool.

<pre>
zkparam.py KERNEL_ELF [options]
</pre>

If you omit options, the tool shows you the current kernel parameters.

<pre>
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf
Current Kernel Parameters:
zepto_console_output=2
</pre>

===Zepto kernel specific kernel parameters===

* '''zepto_debug'''=INTEGER
** Specify the zepto kernel debug level
** 0 turns off all zepto debug msgs. higher number more detail
** default=1
* '''flatmemsizeMB'''=INTEGER
** Specify the size of flatmemory in MB.
** Currently the granularity of memory size is limited to 256 MB
** default=256 min=256 max=1792
* '''zepto_console_output'''=INTEGER
** Specify the console output behavior.
** 0 disables all console output from kernel.
** 1 enables console output from one of nodes(personality rank=1)
** 2 enables console output from all nodes

==Log files,etc==

===Compute Node log===

Debug messages from compute node (i.e, via printk) will appear in one of system log files.
The system log file is recreated every time at system reset. You can find the location of the system log file
by typing the following command.

<pre>
$ ls -1 /bgsys/logs/BGP/sn*-mmcs_db_server*.log|tail -1
/bgsys/logs/BGP/sn1-mmcs_db_server-2009-0209-11:58:20.log
</pre>

Please also take a look at a convenient script BGP/packages/tools/cn-log.sh

===ION Node log===

Debug message from ION node will appear in ION node log files in /bgsys/logs/BGP/,
/bgsys/logs/BGP/R00-M0-N00-J00.log for example.
Each ION has own log file. ION to CN ratio is 1:64 in ANL system. If your job is a 64 (physical) nodes job,
you have one ION log file.

Please also take a look at a convenient script BGP/packages/tools/ion-log.sh

===RAS events===

RAS messages won't appear in the log files.
A command line tool named bg-listevents shows you a record of RAS events.
Type 'bg-listevents -h' for command line parameters.

2009-04-30T03:05:08Z

Kazutomo:

To support high performance computing(HPC) applications, specifically for MPI applications,
we have ported IBM CNK's communication software stack to the Zepto compute node Linux environment.
It is reasonably stable and performance of MPI applications on the Zepto compute node Linux is comparable to that's on CNK.
While there are some limitations on the porting right now, it has some benefits.

Benefits:
* No limitation on the number of thread
** 4 or more openmp job per node
** Additional thread as I/O or backgroup task
* It's on Linux!
** debugging tools such as gdb, strace, etc
** ramfs

Current limitations:
* Only SMP mode is supported
** VN mode will be supported in the future release
* No Binary compatibility between CNK and Zepto CN Linux
** there are slight difference in DCMF and SPI library
* Shared libraries are not provided now
** will be provided in the future release

As in IBM CNK environment, Deep Computing Messaging Framework(DCMF) and System Programming Interface(SPI) are available.
You can also write a DCMF code or a SPI code directly if necessary. DCMF is a communication library that
provides non-blocking operations. Please refer [[http://dcmf.anl-external.org/wiki/index.php/Main_Page DCMF wiki]] for some details. SPI is the lowest level user space API for Torus DMA, collective network, BGP specifc lock mechanisms and
other compute node specific implementations. There is no public document available right now but almost all header files and source codes are available. Internally MPICH depends on DMCF that depends on SPI.

==Compiling HPC applications==

While you can use same compiler to compile your codes,
Zepto compute node environment requires linking with zepto modified libraries.
( MPI application's binary for CNK does not work on Zepto environment ).

===Compilation wrapper scripts===

We provide compilation wrapper scripts (see below) which
automatically links with appropriate libraries
that are installed in your Zepto installation path. We provide the same
set of wrapper scripts that IBM provides. Once you have successfully
compiled your code, you need to submit it with Zepto kernel profile (
see the [[Kernel Profile]] section). Note: only SMP mode is currently supported.

<pre>
- Wrapper scripts that invoke BGP enhanced GNU compilers
zmpicc
zmpicxx
zmpif77
zmpif90

- Wrapper scripts that invoke IBM XL compilers
zmpixlc
zmpixlcxx
zmpixlf2003
zmpixlf77
zmpixlf90
zmpixlf95

- Wrapper scripts that invoke IBM XL compilers(thread safe compilation)
zmpixlc_r
zmpixlcxx_r
zmpixlf2003_r
zmpixlf77_r
zmpixlf90_r
zmpixlf95_r
</pre>

If you need to understand what those script actually do internally, run the wrapper script with the -show option.

===Without compiler scripts===
In case you can't use those compilation wrapper scripts, please make sure
that your makefile or build environemnt points Zepto header files and
libraries correctly. An example would be:

<pre>
/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc \
-o mpi-test-linux -Wall -O3 -I__INST_PREFIX__/include/ mpi-test.c \
-L__INST_PREFIX__/lib/ -lmpich.zcl -ldcmfcoll.zcl -ldcmf.zcl -lSPI.zcl -lzcl \
-lzoid_cn -lrt -lpthread -lm
__INST_PREFIX__/bin/zelftool -e mpi-test-linux
</pre>

'''NOTE:'''
* Replace __INST_PREFIX__ with your actuall Zepto install path
* Don't forget calling the zelftool utility
** which makes your executable a Zepto Compute Binary to let the Zepto kernel load
all application segments into the big memory area.

The file layout in the zepto install path would be:

<pre>
|-- bin
| |-- zelftool
|-- include
| |-- dcmf.h
| |-- dcmf_collectives.h
| |-- dcmf_coremath.h
| |-- dcmf_globalcollectives.h
| |-- dcmf_multisend.h
| |-- dcmf_optimath.h
| |-- mpe_thread.h
| |-- mpi.h
| |-- mpi.mod
| |-- mpi_base.mod
| |-- mpi_constants.mod
| |-- mpi_sizeofs.mod
| |-- mpicxx.h
| |-- mpif.h
| |-- mpio.h
| |-- mpiof.h
| `-- mpix.h
`-- lib
|-- libSPI.zcl.a
|-- libcxxmpich.zcl.a
|-- libdcmf.zcl.a
|-- libdcmfcoll.zcl.a
|-- libfmpich.zcl.a
|-- libfmpich_.zcl.a
|-- libmpich.zcl.a
|-- libmpich.zclf90.a
|-- libzcl.a
`-- libzoid_cn.a
</pre>

==Building MPICH, DCMF and SPI libraries==

We have all necessary source codes to build MPICH, DCMF and SPI.
To build those libraries, just type:

<pre>
$ make -C comm rebuild-target
</pre>

It may take a half hour to an hour to complete the build process, depending on what file system you are using.
i.e., GPFS is definitely slower than local scratch file system.

The rebuild-target target does not know anything about your installation. If you need to apply newly compiled libraries,
do the following steps:

<pre>
$ make -C comm update-prebuilt
$ python install.py __INST_PREFIX__
</pre>

==Software stack layout==

[[Image:Zepto-Comm-Stack.png|right|450px]]

The right figure depicts the layout of communication software stack for Zepto compute node environment.
This is essentially same as in IBM CNK's stack excepts that they have no ZEPTO SPI, and CNK instead of Linux.
While we skip the brief explanation of MPICH since it's well-known software piece,
we briefly describe what DCMF and SPI are here.

* DCMF
** Stands for Deep Computing Messaging Framework
** Developed by IBM originally for BleuGene architecture
** Hardware Initialization, query functions
** Supports BGP Torus DMA, collective network
** Provides timer
** Supports non-blocking collective operations
** BGP MPICH uses DCMF internally (IBM provides a glue layer)
* SPI
** Stands for System Programming Interface
** Developed by IBM. BGP specific codes.
** Kernel interfaces - DMA control, lockbox, etc
** DMA related definitions
*** can be used in both user space and kernel space
** RAS, BGP personality, mapping related functions

BGP SPI is basically designed only for IBM CNK, so SPI is not compatible with Linux.
ZEPTO SPI is a thin software layer that absorbs the differences between CNK and Linux, or drops the requests that Linux can not handle.

==Source code==

The source codes or header files of MPICH, DCMF and SPI can be found in the comm directory.
Technically the source code of MPICH is in the tarball (comm/DCMF/lib/mpich2/mpich2-1.0.7.tar.gz), which will be extracted at build time.

<pre>
|-- DCMF
| |-- lib
| | |-- dev
| | `-- mpich2
| | `-- make
| |-- sys
| | |-- collectives
| | |-- include
| | |-- messaging
|-- arch-runtime
| |-- arch
| | `-- include
| | |-- bpcore
| | |-- cnk
| | |-- common
| | |-- spi
| | `-- zepto
| |-- runtime
| |-- testcodes
| `-- zcl_spi
`-- testcodes
</pre>

MPICH, DCMF, and SPI

2009-04-30T02:37:59Z

Kazutomo:

MPICH, DCMF, and SPI

2009-04-30T02:19:37Z

Kazutomo:

MPICH, DCMF, and SPI

2009-04-30T02:08:18Z

Kazutomo: /* Rebuilding the libraries */

To support high performance computing(HPC) applications, specifically for MPI applications,
we have ported IBM CNK's communication software stack to the Zepto compute node Linux environment.
Performance of MPI applications on the Zepto compute node Linux is comparable to that's on CNK.

As in IBM CNK environment, Deep Computing Messaging Framework(DCMF) and System Programming Interface(SPI) are available.
You can also write a DCMF code or a SPI code directly if necessary. DCMF is a communication library that
provides non-blocking operations. Please refer [[http://dcmf.anl-external.org/wiki/index.php/Main_Page DCMF wiki]] for some details. SPI is the lowest level user space API for Torus DMA, collective network, BGP specifc lock mechanisms and
other compute node specific implementations. There is no public document available right now but almost all header files and source codes are available. Internally MPICH depends on DMCF that depends on SPI.

==Compiling HPC applications==

While you can use same compiler to compile your codes,
Zepto compute node environment requires linking with zepto modified libraries.
( MPI application's binary for CNK does not work on Zepto environment ).

===Compilation wrapper scripts===

We provide compilation wrapper scripts (see below) which
automatically links with appropriate libraries
that are installed in your Zepto installation path. We provide the same
set of wrapper scripts that IBM provides. Once you have successfully
compiled your code, you need to submit it with Zepto kernel profile (
see the [[Kernel Profile]] section). Note: only SMP mode is currently supported.

<pre>
- Wrapper scripts that invoke BGP enhanced GNU compilers
zmpicc
zmpicxx
zmpif77
zmpif90

- Wrapper scripts that invoke IBM XL compilers
zmpixlc
zmpixlcxx
zmpixlf2003
zmpixlf77
zmpixlf90
zmpixlf95

- Wrapper scripts that invoke IBM XL compilers(thread safe compilation)
zmpixlc_r
zmpixlcxx_r
zmpixlf2003_r
zmpixlf77_r
zmpixlf90_r
zmpixlf95_r
</pre>

If you need to understand what those script actually do internally, run the wrapper script with the -show option.

===Without compiler scripts===
In case you can't use those compilation wrapper scripts, please make sure
that your makefile or build environemnt points Zepto header files and
libraries correctly. An example would be:

<pre>
/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc \
-o mpi-test-linux -Wall -O3 -I__INST_PREFIX__/include/ mpi-test.c \
-L__INST_PREFIX__/lib/ -lmpich.zcl -ldcmfcoll.zcl -ldcmf.zcl -lSPI.zcl -lzcl \
-lzoid_cn -lrt -lpthread -lm
__INST_PREFIX__/bin/zelftool -e mpi-test-linux
</pre>

'''NOTE:'''
* Replace __INST_PREFIX__ with your actuall Zepto install path
* Don't forget calling the zelftool utility
** which makes your executable a Zepto Compute Binary to let the Zepto kernel load
all application segments into the big memory area.

The file layout in the zepto install path would be:

<pre>
|-- bin
| |-- zelftool
|-- include
| |-- dcmf.h
| |-- dcmf_collectives.h
| |-- dcmf_coremath.h
| |-- dcmf_globalcollectives.h
| |-- dcmf_multisend.h
| |-- dcmf_optimath.h
| |-- mpe_thread.h
| |-- mpi.h
| |-- mpi.mod
| |-- mpi_base.mod
| |-- mpi_constants.mod
| |-- mpi_sizeofs.mod
| |-- mpicxx.h
| |-- mpif.h
| |-- mpio.h
| |-- mpiof.h
| `-- mpix.h
`-- lib
|-- libSPI.zcl.a
|-- libcxxmpich.zcl.a
|-- libdcmf.zcl.a
|-- libdcmfcoll.zcl.a
|-- libfmpich.zcl.a
|-- libfmpich_.zcl.a
|-- libmpich.zcl.a
|-- libmpich.zclf90.a
|-- libzcl.a
`-- libzoid_cn.a
</pre>

==Building MPICH, DCMF and SPI libraries==

We have all necessary source codes to build MPICH, DCMF and SPI.
To build those libraries, just type:

<pre>
$ make -C comm rebuild-target
</pre>

It may take a half hour to an hour to complete the build process, depending on what file system you are using.
i.e., GPFS is definitely slower than local scratch file system.

The rebuild-target target does not know anything about your installation. If you need to apply newly compiled libraries,
do the following steps:

<pre>
$ make -C comm update-prebuilt
$ python install.py __INST_PREFIX__
</pre>

==Software stack layout==

[[Image:Zepto-Comm-Stack.png|right|450px]]

The right figure depicts the layout of communication software stack for Zepto compute node environment.
This is essentially same as in IBM CNK's stack excepts that they have no ZEPTO SPI, and CNK instead of Linux.
While we skip the brief explanation of MPICH since it's well-known software piece,
we briefly describe what DCMF and SPI are here.

* DCMF
** Stands for Deep Computing Messaging Framework
** Developed by IBM originally for BleuGene architecture
** Hardware Initialization, query functions
** Supports BGP Torus DMA, collective network
** Provides timer
** Supports non-blocking collective operations
** BGP MPICH uses DCMF internally (IBM provides a glue layer)
* SPI
** Stands for System Programming Interface
** Developed by IBM. BGP specific codes.
** Kernel interfaces - DMA control, lockbox, etc
** DMA related definitions
*** can be used in both user space and kernel space
** RAS, BGP personality, mapping related functions

BGP SPI is basically designed only for IBM CNK, so SPI is not compatible with Linux.
ZEPTO SPI is a thin software layer that absorbs the differences between CNK and Linux, or drops the requests that Linux can not handle.

==Source code==

The source codes or header files of MPICH, DCMF and SPI can be found in the comm directory.
Technically the source code of MPICH is in the tarball (comm/DCMF/lib/mpich2/mpich2-1.0.7.tar.gz), which will be extracted at build time.

<pre>
|-- DCMF
| |-- lib
| | |-- dev
| | `-- mpich2
| | `-- make
| |-- sys
| | |-- collectives
| | |-- include
| | |-- messaging
|-- arch-runtime
| |-- arch
| | `-- include
| | |-- bpcore
| | |-- cnk
| | |-- common
| | |-- spi
| | `-- zepto
| |-- runtime
| |-- testcodes
| `-- zcl_spi
`-- testcodes
</pre>

MPICH, DCMF, and SPI

2009-04-30T02:06:36Z

Kazutomo:

To support high performance computing(HPC) applications, specifically for MPI applications,
we have ported IBM CNK's communication software stack to the Zepto compute node Linux environment.
Performance of MPI applications on the Zepto compute node Linux is comparable to that's on CNK.

As in IBM CNK environment, Deep Computing Messaging Framework(DCMF) and System Programming Interface(SPI) are available.
You can also write a DCMF code or a SPI code directly if necessary. DCMF is a communication library that
provides non-blocking operations. Please refer [[http://dcmf.anl-external.org/wiki/index.php/Main_Page DCMF wiki]] for some details. SPI is the lowest level user space API for Torus DMA, collective network, BGP specifc lock mechanisms and
other compute node specific implementations. There is no public document available right now but almost all header files and source codes are available. Internally MPICH depends on DMCF that depends on SPI.

==Compiling HPC applications==

While you can use same compiler to compile your codes,
Zepto compute node environment requires linking with zepto modified libraries.
( MPI application's binary for CNK does not work on Zepto environment ).

===Compilation wrapper scripts===

We provide compilation wrapper scripts (see below) which
automatically links with appropriate libraries
that are installed in your Zepto installation path. We provide the same
set of wrapper scripts that IBM provides. Once you have successfully
compiled your code, you need to submit it with Zepto kernel profile (
see the [[Kernel Profile]] section). Note: only SMP mode is currently supported.

<pre>
- Wrapper scripts that invoke BGP enhanced GNU compilers
zmpicc
zmpicxx
zmpif77
zmpif90

- Wrapper scripts that invoke IBM XL compilers
zmpixlc
zmpixlcxx
zmpixlf2003
zmpixlf77
zmpixlf90
zmpixlf95

- Wrapper scripts that invoke IBM XL compilers(thread safe compilation)
zmpixlc_r
zmpixlcxx_r
zmpixlf2003_r
zmpixlf77_r
zmpixlf90_r
zmpixlf95_r
</pre>

If you need to understand what those script actually do internally, run the wrapper script with the -show option.

===Without compiler scripts===
In case you can't use those compilation wrapper scripts, please make sure
that your makefile or build environemnt points Zepto header files and
libraries correctly. An example would be:

<pre>
/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc \
-o mpi-test-linux -Wall -O3 -I__INST_PREFIX__/include/ mpi-test.c \
-L__INST_PREFIX__/lib/ -lmpich.zcl -ldcmfcoll.zcl -ldcmf.zcl -lSPI.zcl -lzcl \
-lzoid_cn -lrt -lpthread -lm
__INST_PREFIX__/bin/zelftool -e mpi-test-linux
</pre>

'''NOTE:'''
* Replace __INST_PREFIX__ with your actuall Zepto install path
* Don't forget calling the zelftool utility
** which makes your executable a Zepto Compute Binary to let the Zepto kernel load
all application segments into the big memory area.

The file layout in the zepto install path would be:

<pre>
|-- bin
| |-- zelftool
|-- include
| |-- dcmf.h
| |-- dcmf_collectives.h
| |-- dcmf_coremath.h
| |-- dcmf_globalcollectives.h
| |-- dcmf_multisend.h
| |-- dcmf_optimath.h
| |-- mpe_thread.h
| |-- mpi.h
| |-- mpi.mod
| |-- mpi_base.mod
| |-- mpi_constants.mod
| |-- mpi_sizeofs.mod
| |-- mpicxx.h
| |-- mpif.h
| |-- mpio.h
| |-- mpiof.h
| `-- mpix.h
`-- lib
|-- libSPI.zcl.a
|-- libcxxmpich.zcl.a
|-- libdcmf.zcl.a
|-- libdcmfcoll.zcl.a
|-- libfmpich.zcl.a
|-- libfmpich_.zcl.a
|-- libmpich.zcl.a
|-- libmpich.zclf90.a
|-- libzcl.a
`-- libzoid_cn.a
</pre>

==Rebuilding the libraries==

We have all necessary source codes to build MPICH, DCMF and SPI.
To build those libraries, just type:

<pre>
$ make -C comm rebuild-target
</pre>

It may take a half hour to an hour to complete the build process, depending on what file system you are using.
i.e., GPFS is definitely slower than local scratch file system.

The rebuild-target target does not know anything about your installation. If you need to apply newly compiled libraries,
do the following steps:

<pre>
$ make -C comm update-prebuilt
$ python install.py __INST_PREFIX__
</pre>

==Software stack layout==

[[Image:Zepto-Comm-Stack.png|right|450px]]

The right figure depicts the layout of communication software stack for Zepto compute node environment.
This is essentially same as in IBM CNK's stack excepts that they have no ZEPTO SPI, and CNK instead of Linux.
While we skip the brief explanation of MPICH since it's well-known software piece,
we briefly describe what DCMF and SPI are here.

* DCMF
** Stands for Deep Computing Messaging Framework
** Developed by IBM originally for BleuGene architecture
** Hardware Initialization, query functions
** Supports BGP Torus DMA, collective network
** Provides timer
** Supports non-blocking collective operations
** BGP MPICH uses DCMF internally (IBM provides a glue layer)
* SPI
** Stands for System Programming Interface
** Developed by IBM. BGP specific codes.
** Kernel interfaces - DMA control, lockbox, etc
** DMA related definitions
*** can be used in both user space and kernel space
** RAS, BGP personality, mapping related functions

BGP SPI is basically designed only for IBM CNK, so SPI is not compatible with Linux.
ZEPTO SPI is a thin software layer that absorbs the differences between CNK and Linux, or drops the requests that Linux can not handle.

==Source code==

The source codes or header files of MPICH, DCMF and SPI can be found in the comm directory.
Technically the source code of MPICH is in the tarball (comm/DCMF/lib/mpich2/mpich2-1.0.7.tar.gz), which will be extracted at build time.

<pre>
|-- DCMF
| |-- lib
| | |-- dev
| | `-- mpich2
| | `-- make
| |-- sys
| | |-- collectives
| | |-- include
| | |-- messaging
|-- arch-runtime
| |-- arch
| | `-- include
| | |-- bpcore
| | |-- cnk
| | |-- common
| | |-- spi
| | `-- zepto
| |-- runtime
| |-- testcodes
| `-- zcl_spi
`-- testcodes
</pre>