Installation
Configuration | Top | Testing
Installing the support files
Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) use primarily when building compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and type:
$ python install.py /path/to/install
This will install the following files:
Install Home: /path/to/install Creating install directories ... Making /path/to/install/bin Making /path/to/install/include Making /path/to/install/lib Installing mpich2 scripts ... Installing /path/to/install/bin/zmpicc Installing /path/to/install/bin/zmpicxx Installing /path/to/install/bin/zmpif77 Installing /path/to/install/bin/zmpif90 Installing /path/to/install/bin/zmpixlc_r Installing /path/to/install/bin/zmpixlc Installing /path/to/install/bin/zmpixlcxx_r Installing /path/to/install/bin/zmpixlcxx Installing /path/to/install/bin/zmpixlf2003_r Installing /path/to/install/bin/zmpixlf2003 Installing /path/to/install/bin/zmpixlf77_r Installing /path/to/install/bin/zmpixlf77 Installing /path/to/install/bin/zmpixlf90_r Installing /path/to/install/bin/zmpixlf90 Installing /path/to/install/bin/zmpixlf95_r Installing /path/to/install/bin/zmpixlf95 Installing misc files ... /path/to/install/bin/zelftool Installing MPICH2 headers ... /path/to/install/include/mpe_thread.h /path/to/install/include/mpi_base.mod /path/to/install/include/mpi_constants.mod /path/to/install/include/mpicxx.h /path/to/install/include/mpif.h /path/to/install/include/mpi.h /path/to/install/include/mpi.mod /path/to/install/include/mpiof.h /path/to/install/include/mpio.h /path/to/install/include/mpi_sizeofs.mod /path/to/install/include/mpix.h Installing MPICH2 libraries ... /path/to/install/lib/libcxxmpich.zcl.a /path/to/install/lib/libfmpich_.zcl.a /path/to/install/lib/libfmpich.zcl.a /path/to/install/lib/libmpich.zcl.a /path/to/install/lib/libmpich.zclf90.a Installing SPI libraries ... /path/to/install/lib/libSPI.zcl.a /path/to/install/lib/libzcl.a Installing Zoid files ... /path/to/install/lib/libzoid_cn.a Installing DCMF files ... /path/to/install/include/dcmf_collectives.h /path/to/install/include/dcmf_coremath.h /path/to/install/include/dcmf_globalcollectives.h /path/to/install/include/dcmf.h /path/to/install/include/dcmf_multisend.h /path/to/install/include/dcmf_optimath.h /path/to/install/lib/libdcmfcoll.zcl.a /path/to/install/lib/libdcmf.zcl.a
Setting up a kernel profile
The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.
Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units. More importantly, the system supports using different software stacks (what we call kernel profiles) on different partitions at the same time. This means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.
A kernel profile consists of the following elements:
- loader
- A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
- CN image list
- A series of images loaded onto each compute node.
- ION image list
- A series of images loaded onto each I/O node.
All the images are loaded onto the machine from the service node via the service (JTAG) network. The loader is loaded first, followed by the CN- and ION-specific images loaded in order.
The CN image list defaults to CNS followed by the CNK. The ION image list defaults to CNS, followed by the Linux kernel, followed by the Linux ramdisk. CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.
To enable ZeptoOS, we need to boot the Zepto CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.
System using Cobalt
If the BGP system in question has the Cobalt scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS there should be easy. All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.
On Argonne BGP machines, /bgsys/argonne-utils/profiles/ is the top-level kernel profile directory; individual profiles are stored in its subdirectories. Assuming that the ZeptoOS images have already been created (see Configuration), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:
$ cd <kernel_profile_dir> $ mkdir <your_profile> && cd <your_profile> $ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK $ ln -s <zepto_dir>/BGP-ION-zImage.elf INK $ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk $ ln -s ../default/CNS $ ln -s ../default/uloader
Note: ensure that the ZeptoOS images are world-readable, otherwise jobs will fail to start. If needed, copy the images to the kernel profile rather than link to them.
For Argonne users, we provide a convenience script mkprofile-ANL.sh with some extra features. The following command line is equivalent to the above steps:
$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>
Invoking it with the -h option prints an overview of command line options. In particular, use -c if you prefer to copy images instead of making the links.
$ ./mkprofile-ANL.sh -h Usage: ./mkprofile-ANL.sh [OPTIONS] Options: -h : Show this message -c : Copy images instead of making symbolic link -f : Overwrite existing profile --profile=name : Specify profile name --cn=fn : Compute Node Kernel Image --ion=fn : Specify I/O Node Kernel Image --rd=fn : Specify I/O Node Ramdisk Image --ls : show files in profile --dryrun
Once the ZeptoOS kernel profile is set up, it can be booted by specifying the profile name when submitting a job using either cqsub or qsub:
$ cqsub -k <profile_name> ... $ qsub --kernel <profile_name> ...
Manual installation using the MMCS console
Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed. If you don't have cobalt, probably we'll need to use Midplane Management Control System(MMCS). We explain how to assign and boot your own kernel images using MMCS.
Here is a brief summary of MMCS.
- Available on all BGP systems
- the lowest control mechanism for BGP partition
- allocate, free or query of block(partition)
- status check
- assign boot images
- low level debug command
Due to its low level interface, it requires administrator level permission to use it. You also need to reserve a partition (or block).
Assign Zepto images to a BGP partition
Login to the service node and start MMCS.
$ ssh sn sn $ ./mmcs.sh
[mmcs.sh]
#!/bin/sh
export DB2HOME=/dbhome/bgpdb2c/sqllib
DB2SRC=${DB2HOME}/db2profile
[ -f "$DB2SRC" ] && . $DB2SRC
cd /bgsys/drivers/ppcfloor/bin
./mmcs_db_console
Please memorize that the current configuration. You need to revert the blockinfo to the original configuration later on after you are done with Zepto kernel.
console $ set_username YOUR_LOGIN_NAME console $ getblockinfo BGP_BLOCK_NAME OK boot info for block BGP_BLOCK_NAME: mloader: /bgsys/drivers/ppcfloor/boot/uloader cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk status: F
Assign Zepto images to a parition
console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,BGP_CN_LINUX_KERNEL_PATH /bgsys/drivers/ppcfloor\ /boot/cns,BGP_ION_LINUX_KERNEL_PATH,BGP_ION_LINUX_RAMDISK_PATH console $ quit
Boot Zepto kernel
Once you have configured a partition with Zepto kernels correctly, Zepto kernels will be booted when you run a job on that partition(via mpirun for example)
fen $ mpirun -verbose 1 -partition BGP_BLOCK_NAME -np 64 -timeout 600 -cwd `pwd` -exe ./a.out
Restore to the original configuration(Don't forget!!!)
After you have done your work on Zepto kernel, you need to restore to the original configuration. Here is an example.
fen $ ssh sn sn $ ./mmcs.sh console $ set_username YOUR_LOGIN_NAME console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk /bgsys/drivers/\ ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk console $ quit
Configuration | Top | Testing
