Installation
Configuration | Top | Testing
Installing the support files
Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) used primarily when building compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and type:
$ python install.py /path/to/install
This will install the following files:
Install Home: /path/to/install Creating install directories ... Making /path/to/install/bin Making /path/to/install/cnbin Making /path/to/install/include Making /path/to/install/lib Making /path/to/install/lib/zoid Installing MPICH2 scripts ... /path/to/install/bin/zmpicc /path/to/install/bin/zmpicxx /path/to/install/bin/zmpif77 /path/to/install/bin/zmpif90 /path/to/install/bin/zmpixlc /path/to/install/bin/zmpixlc_r /path/to/install/bin/zmpixlcxx /path/to/install/bin/zmpixlcxx_r /path/to/install/bin/zmpixlf2003 /path/to/install/bin/zmpixlf2003_r /path/to/install/bin/zmpixlf77 /path/to/install/bin/zmpixlf77_r /path/to/install/bin/zmpixlf90 /path/to/install/bin/zmpixlf90_r /path/to/install/bin/zmpixlf95 /path/to/install/bin/zmpixlf95_r Installing Compute Node binaries ... /path/to/install/cnbin/cn-ipfwd.sh /path/to/install/cnbin/cn-ipfwd Installing misc files ... /path/to/install/bin/zelftool /path/to/install/bin/zkparam.py Installing MPICH2 headers ... /path/to/install/include/mpe_thread.h /path/to/install/include/mpi.h /path/to/install/include/mpi.mod /path/to/install/include/mpi_base.mod /path/to/install/include/mpi_constants.mod /path/to/install/include/mpi_sizeofs.mod /path/to/install/include/mpicxx.h /path/to/install/include/mpif.h /path/to/install/include/mpio.h /path/to/install/include/mpiof.h /path/to/install/include/mpix.h Installing MPICH2 libraries ... /path/to/install/lib/libcxxmpich.zcl.a /path/to/install/lib/libfmpich.zcl.a /path/to/install/lib/libfmpich_.zcl.a /path/to/install/lib/libmpich.zcl.a /path/to/install/lib/libmpich.zclf90.a Installing SPI libraries ... /path/to/install/lib/libSPI.zcl.a /path/to/install/lib/libzcl.a Installing ZOID files ... /path/to/install/lib/libzoid_cn.a /path/to/install/lib/zoid/libc.a /path/to/install/lib/zoid/libpthread.a Installing DCMF files ... /path/to/install/include/dcmf.h /path/to/install/include/dcmf_collectives.h /path/to/install/include/dcmf_coremath.h /path/to/install/include/dcmf_globalcollectives.h /path/to/install/include/dcmf_multisend.h /path/to/install/include/dcmf_optimath.h /path/to/install/lib/libdcmf.zcl.a /path/to/install/lib/libdcmfcoll.zcl.a
Setting up a kernel profile
The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.
Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units. More importantly, the system supports using different software stacks (what we call kernel profiles) on different partitions at the same time. For our immediate purposes, this means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.
A kernel profile consists of the following elements:
- loader
- A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
- CN image list
- A series of images loaded onto each compute node.
- ION image list
- A series of images loaded onto each I/O node.
All the images are loaded onto the machine from the service node via the service (JTAG) network. The loader is loaded first, followed by the CN- and ION-specific images loaded in order.
The CN image list defaults to the CNS followed by the CNK. The ION image list defaults to the CNS, followed by the Linux kernel, followed by the Linux ramdisk. CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.
To enable ZeptoOS, we need to boot the ZeptoOS CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.
System using Cobalt
If the BGP system in question has the Cobalt scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS should be easy. All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.
On Argonne BGP machines, /bgsys/argonne-utils/profiles/ is the top-level kernel profile directory; individual profiles are stored in its subdirectories. Assuming that the ZeptoOS images have already been created (see Configuration), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:
$ cd <kernel_profile_dir> $ mkdir <your_profile> && cd <your_profile> $ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK $ ln -s <zepto_dir>/BGP-ION-zImage.elf INK $ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk $ ln -s ../default/CNS $ ln -s ../default/uloader
Note: ensure that the ZeptoOS images are world-readable from the service node, otherwise jobs will fail to start. If needed, copy the images to the kernel profile rather than link to them.
For Argonne users, we provide a convenience script mkprofile-ANL.sh with some extra features. The following command line is equivalent to the above steps:
$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>
Invoking the script with the -h option prints an overview of command line options. In particular, use -c if you prefer to copy images instead of making the links.
$ ./mkprofile-ANL.sh -h Usage: ./mkprofile-ANL.sh [OPTIONS] Options: -h : Show this message -c : Copy images instead of making symbolic link -f : Overwrite existing profile --profile=name : Specify profile name --cn=fn : Compute Node Kernel Image --ion=fn : Specify I/O Node Kernel Image --rd=fn : Specify I/O Node Ramdisk Image --ls : show files in profile --dryrun
Once the ZeptoOS kernel profile is set up, it can be booted by specifying the profile name when submitting a job using either cqsub or qsub:
$ cqsub -k <profile_name> ... $ qsub --kernel <profile_name> ...
Testing and troubleshooting ZeptoOS is discussed in detail in the next section.
Manual installation using the MMCS console
If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS). MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.
Before we begin, please select a partition to experiment on. The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).
Assigning ZeptoOS images to the partition
Log on the service node and start the MMCS console:
$ ssh <service_node> sn $ mmcs_db_console connecting to mmcs_server connected to mmcs_server connected to DB2 mmcs$
We begin by obtaining the current boot configuration of the partition, so that at the end we can revert it to the initial condition:
mmcs$ getblockinfo <partition_name> OK boot info for block <partition_name>: mloader: /bgsys/drivers/ppcfloor/boot/uloader cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk status: F mmcs$
Now assign ZeptoOS images to the partition:
mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \ /bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-CN-zImage-with-initrd.elf \ /bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf mmcs$ quit
Booting ZeptoOS
Once the partition has been correctly configured, the ZeptoOS images will be loaded when you run a job on that partition (via mpirun, for example):
$ mpirun -verbose 1 -partition <partition_name> -np 64 -timeout 600 -cwd $PWD -exe ./a.out
Testing and troubleshooting ZeptoOS is discussed in detail in the next section.
Restore the original configuration
Do not forget this step!
Once the experiments are over, restore the partition to its original configuration:
$ ssh <service_node> sn $ mmcs_db_console connecting to mmcs_server connected to mmcs_server connected to DB2 mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \ /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk \ /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk mmcs$ quit
Configuration | Top | Testing
