Installation
Configuration | Top | Testing
Installing the support files
Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) needed to build compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and just type:
$ python install.py /path/to/install
This will install the following files:
Install Home: /path/to/install Creating install directories ... Making /path/to/install/bin Making /path/to/install/include Making /path/to/install/lib Installing mpich2 scripts ... Installing /path/to/install/bin/zmpicc Installing /path/to/install/bin/zmpicxx Installing /path/to/install/bin/zmpif77 Installing /path/to/install/bin/zmpif90 Installing /path/to/install/bin/zmpixlc_r Installing /path/to/install/bin/zmpixlc Installing /path/to/install/bin/zmpixlcxx_r Installing /path/to/install/bin/zmpixlcxx Installing /path/to/install/bin/zmpixlf2003_r Installing /path/to/install/bin/zmpixlf2003 Installing /path/to/install/bin/zmpixlf77_r Installing /path/to/install/bin/zmpixlf77 Installing /path/to/install/bin/zmpixlf90_r Installing /path/to/install/bin/zmpixlf90 Installing /path/to/install/bin/zmpixlf95_r Installing /path/to/install/bin/zmpixlf95 Installing misc files ... /path/to/install/bin/zelftool Installing MPICH2 headers ... /path/to/install/include/mpe_thread.h /path/to/install/include/mpi_base.mod /path/to/install/include/mpi_constants.mod /path/to/install/include/mpicxx.h /path/to/install/include/mpif.h /path/to/install/include/mpi.h /path/to/install/include/mpi.mod /path/to/install/include/mpiof.h /path/to/install/include/mpio.h /path/to/install/include/mpi_sizeofs.mod /path/to/install/include/mpix.h Installing MPICH2 libraries ... /path/to/install/lib/libcxxmpich.zcl.a /path/to/install/lib/libfmpich_.zcl.a /path/to/install/lib/libfmpich.zcl.a /path/to/install/lib/libmpich.zcl.a /path/to/install/lib/libmpich.zclf90.a Installing SPI libraries ... /path/to/install/lib/libSPI.zcl.a /path/to/install/lib/libzcl.a Installing Zoid files ... /path/to/install/lib/libzoid_cn.a Installing DCMF files ... /path/to/install/include/dcmf_collectives.h /path/to/install/include/dcmf_coremath.h /path/to/install/include/dcmf_globalcollectives.h /path/to/install/include/dcmf.h /path/to/install/include/dcmf_multisend.h /path/to/install/include/dcmf_optimath.h /path/to/install/lib/libdcmfcoll.zcl.a /path/to/install/lib/libdcmf.zcl.a
Setting up a kernel profile
The BlueGene/P system is capble of assigning different boot images per partition. The system allows us to specify loader program, CN images and ION images. The loader program is loaded into main memory via jtag network and executed first. /bgsys/drivers/ppcfloor/boot/uloader is only loader program, which is not an open source code. The system allows us to specify multiple CN images. Command Node Service(CNS) image and CN kernel image(IBM CNK) are default images and loaded into CN's main memory in order. /bgsys/drivers/ppcfloor/boot/cns is only choice for CNS, which is not an open source code , either. We can also specify multiple ION images. CNS, IBM ION Linux kernel image and ION Linux ramdisk are default images. They are also loaded into ION's main memory in order.
To enable Zepto feature, we need to boot Zepto CN kernel and ION kernel in a partition that we use. We describe how to assign and boot Zepto images in this section.
Cobalt installed system
If your BGP system has the cobalt scheduler installed and its kernel profile feature has been configured properly, it would be easy to boot Zepto kernel for your task.
What you need is to make a directory in the kernel profile directory and create a couple of symbolic link that point to Zepto images.
In ANL BGP, /bgsys/argonne-utils/profiles/ is the kernel profile directory. Here are the concrete steps to create a new kernel profile. Suppose that you have already built your Zepto kernel images and write permission to the kernel profile directory.
$ cd KERNEL_PROFILE_DIR $ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME $ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf CNK $ ln -s ZEPTO_DIR/BGP-ION-zImage.elf INK $ ln -s ZEPTO_DIR/BGP-ION-ramdisk-for-CNL.elf ramdisk $ ln -s ../factory-default/CNS $ ln -s ../factory-default/uloader
NOTE: your Zepto images must be readable from others, otherwise your job will fail. Please double check!!!
For ANL user, we provide a convenient script named mkprofile-ANL.sh which essentially does what mentioned in above but has some extra features. The following command line is equivalent to the steps described in above.
$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME
Invoking it with the -h option shows its help message. Use -c if you actually need to copy images instead of making symbolic link. Use -cn, -ion or -rd if you have a custom named image.
$ ./mkprofile-ANL.sh -h Usage: ./mkprofile-ANL.sh [OPTIONS] Options: -h : Show this message -c : Copy images instead of making symbolic link -f : Overwrite existing profile --profile=name : Specify profile name --cn=fn : Compute Node Kernel Image --ion=fn : Specify I/O Node Kernel Image --rd=fn : Specify I/O Node Ramdisk Image --ls : show files in profile --dryrun
Once you have properly setup your Zepto kernel profile, you can boot Zepto kernel by specifying your kernel profile name via the -k cobalt option.
$ cqsub -k YOUR_PROFILE_NAME ....
MMCS console
Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed. If you don't have cobalt, probably we'll need to use Midplane Management Control System(MMCS). We explain how to assign and boot your own kernel images using MMCS.
Here is a brief summary of MMCS.
- Available on all BGP systems
- the lowest control mechanism for BGP partition
- allocate, free or query of block(partition)
- status check
- assign boot images
- low level debug command
Due to its low level interface, it requires administrator level permission to use it. You also need to reserve a partition (or block).
Assign Zepto images to a BGP partition
Login to the service node and start MMCS.
$ ssh sn sn $ ./mmcs.sh
[mmcs.sh]
#!/bin/sh
export DB2HOME=/dbhome/bgpdb2c/sqllib
DB2SRC=${DB2HOME}/db2profile
[ -f "$DB2SRC" ] && . $DB2SRC
cd /bgsys/drivers/ppcfloor/bin
./mmcs_db_console
Please memorize that the current configuration. You need to revert the blockinfo to the original configuration later on after you are done with Zepto kernel.
console $ set_username YOUR_LOGIN_NAME console $ getblockinfo BGP_BLOCK_NAME OK boot info for block BGP_BLOCK_NAME: mloader: /bgsys/drivers/ppcfloor/boot/uloader cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk status: F
Assign Zepto images to a parition
console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,BGP_CN_LINUX_KERNEL_PATH /bgsys/drivers/ppcfloor\ /boot/cns,BGP_ION_LINUX_KERNEL_PATH,BGP_ION_LINUX_RAMDISK_PATH console $ quit
Boot Zepto kernel
Once you have configured a partition with Zepto kernels correctly, Zepto kernels will be booted when you run a job on that partition(via mpirun for example)
fen $ mpirun -verbose 1 -partition BGP_BLOCK_NAME -np 64 -timeout 600 -cwd `pwd` -exe ./a.out
Restore to the original configuration(Don't forget!!!)
After you have done your work on Zepto kernel, you need to restore to the original configuration. Here is an example.
fen $ ssh sn sn $ ./mmcs.sh console $ set_username YOUR_LOGIN_NAME console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk /bgsys/drivers/\ ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk console $ quit
Configuration | Top | Testing
