Installation

Configuration | Top | Testing

---

Installing the support files

Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) use primarily when building compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and type:

$ python install.py /path/to/install

This will install the following files:

Install Home:  /path/to/install
Creating install directories ...
Making /path/to/install/bin
Making /path/to/install/include
Making /path/to/install/lib

Installing mpich2 scripts ...
Installing /path/to/install/bin/zmpicc
Installing /path/to/install/bin/zmpicxx
Installing /path/to/install/bin/zmpif77
Installing /path/to/install/bin/zmpif90
Installing /path/to/install/bin/zmpixlc_r
Installing /path/to/install/bin/zmpixlc
Installing /path/to/install/bin/zmpixlcxx_r
Installing /path/to/install/bin/zmpixlcxx
Installing /path/to/install/bin/zmpixlf2003_r
Installing /path/to/install/bin/zmpixlf2003
Installing /path/to/install/bin/zmpixlf77_r
Installing /path/to/install/bin/zmpixlf77
Installing /path/to/install/bin/zmpixlf90_r
Installing /path/to/install/bin/zmpixlf90
Installing /path/to/install/bin/zmpixlf95_r
Installing /path/to/install/bin/zmpixlf95

Installing misc files ...
/path/to/install/bin/zelftool

Installing MPICH2 headers ...
/path/to/install/include/mpe_thread.h
/path/to/install/include/mpi_base.mod
/path/to/install/include/mpi_constants.mod
/path/to/install/include/mpicxx.h
/path/to/install/include/mpif.h
/path/to/install/include/mpi.h
/path/to/install/include/mpi.mod
/path/to/install/include/mpiof.h
/path/to/install/include/mpio.h
/path/to/install/include/mpi_sizeofs.mod
/path/to/install/include/mpix.h

Installing MPICH2 libraries ...
/path/to/install/lib/libcxxmpich.zcl.a
/path/to/install/lib/libfmpich_.zcl.a
/path/to/install/lib/libfmpich.zcl.a
/path/to/install/lib/libmpich.zcl.a
/path/to/install/lib/libmpich.zclf90.a

Installing SPI libraries ...
/path/to/install/lib/libSPI.zcl.a
/path/to/install/lib/libzcl.a

Installing Zoid files ...
/path/to/install/lib/libzoid_cn.a

Installing DCMF files ...
/path/to/install/include/dcmf_collectives.h
/path/to/install/include/dcmf_coremath.h
/path/to/install/include/dcmf_globalcollectives.h
/path/to/install/include/dcmf.h
/path/to/install/include/dcmf_multisend.h
/path/to/install/include/dcmf_optimath.h
/path/to/install/lib/libdcmfcoll.zcl.a
/path/to/install/lib/libdcmf.zcl.a

Setting up a kernel profile

The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.

Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units. More importantly, the system supports using different software stacks (what we call kernel profiles) on different partitions at the same time. This means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.

A kernel profile consists of the following elements:

loader

A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.

CN image list

A series of images loaded onto each compute node.

ION image list

A series of images loaded onto each I/O node.

All the images are loaded onto the machine from the service node via the service (JTAG) network. The loader is loaded first, followed by the CN- and ION-specific images loaded in order.

The CN image list defaults to CNS followed by the CNK. The ION image list defaults to CNS, followed by the Linux kernel, followed by the Linux ramdisk. CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.

To enable ZeptoOS, we need to boot the Zepto CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.

In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.

System using Cobalt

If the BGP system in question has the Cobalt scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS there should be easy. All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.

On Argonne BGP machines, /bgsys/argonne-utils/profiles/ is the top-level kernel profile directory; individual profiles are stored in its subdirectories. Assuming that the ZeptoOS images have already been created (see Configuration), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:

$ cd <kernel_profile_dir>
$ mkdir <your_profile> && cd <your_profile>
$ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK
$ ln -s <zepto_dir>/BGP-ION-zImage.elf INK
$ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk
$ ln -s ../default/CNS
$ ln -s ../default/uloader

Note: ensure that the ZeptoOS images are world-readable, otherwise jobs will fail to start. If needed, copy the images to the kernel profile rather than link to them.

For Argonne users, we provide a convenience script mkprofile-ANL.sh with some extra features. The following command line is equivalent to the above steps:

$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>

Invoking it with the -h option prints an overview of command line options. In particular, use -c if you prefer to copy images instead of making the links.

$ ./mkprofile-ANL.sh -h
Usage: ./mkprofile-ANL.sh [OPTIONS]

Options:
-h             : Show this message
-c             : Copy images instead of making symbolic link
-f             : Overwrite existing profile
--profile=name : Specify profile name
--cn=fn        : Compute Node Kernel Image
--ion=fn       : Specify I/O Node Kernel Image
--rd=fn        : Specify I/O Node Ramdisk Image
--ls           : show files in profile
--dryrun

Once the ZeptoOS kernel profile is set up, it can be booted by specifying the profile name when submitting a job using either cqsub or qsub:

$ cqsub -k <profile_name> ...
$ qsub --kernel <profile_name> ...

Testing and troubleshooting ZeptoOS is discussed in detail in the next section.

Manual installation using the MMCS console

If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS). MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.

Before we begin, please select a partition that we can experiment on. The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).

Assigning ZeptoOS images to the partition

Log on the service node and start the MMCS console:

$ ssh <service_node>
sn $ mmcs_db_console
connecting to mmcs_server
connected to mmcs_server
connected to DB2
mmcs$

We begin by obtaining the current configuration of a partition, so that at the end we can revert it to the initial condition:

mmcs$ getblockinfo <partition_name>
OK
boot info for block <partition_name>:
mloader: /bgsys/drivers/ppcfloor/boot/uloader
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
status: F
mmcs$ 

Now assign ZeptoOS images to the partition:

mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-CN-zImage-with-initrd.elf \
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf
mmcs$ quit

Booting ZeptoOS

Once you have correctly configured a partition, the ZeptoOS images will be loaded when you run a job on that partition (via mpirun, for example):

$ mpirun -verbose 1 -partition <partition_name> -np 64 -timeout 600 -cwd $PWD -exe ./a.out

Testing and troubleshooting ZeptoOS is discussed in detail in the next section.

Restore the original configuration

Do not forget this step!

After you are finished using ZeptoOS, you need to restore the partition to its original configuration:

$ ssh <service_node>
sn $ mmcs_db_console
connecting to mmcs_server
connected to mmcs_server
connected to DB2
mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk \
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
mmcs$ quit

---

Configuration | Top | Testing