Kernel Profile

Introduction

The BlueGene/P system is capble of assigned different boot images per partition. The system allows us to specify loader program, CN images and ION images. The loader program is loaded into main memory via jtag network and executed first. /bgsys/drivers/ppcfloor/boot/uloader is only choice for the loader program. No source code of uloader is available. We can can specify multiple CN images. By default, Command Node Service(CNS) image and CN kernel image(IBM CNK) are specified. They are loaded into main memory in order. /bgsys/drivers/ppcfloor/boot/cns is only choice for CNS. No source code of cns is available. We can also specify multiple ION images. By default, CNS, IBM ION Linux kernel image and ION Linux ramdisk are specified. They are also loaded in order.

To enable Zepto feature, we need to boot Zepto CN kernel and ION kernel in a partition that we use. We describe how to assign and boot Zepto images in this section.

Cobalt installed system

If your BGP system has the cobalt scheduler installed and its kernel profile feature has been configured properly, it would be easy to boot Zepto kernel for your computational job.

What you need to make a directory in the kernel profile directory and a create a couple of symbolic link that point to Zepto images. In ANL BGP, /bgsys/argonne-utils/profiles/ is the kernel profile directory. Here are concrete steps to create a new kernel profile. Suppose that you have already built your Zepto kernel images and write permission to the kernel profile directory.

$ cd KERNEL_PROFILE_DIR 
$ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME 
$ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf  CNK 
$ ln -s ZEPTO_DIR/BGP-ION-zImage.elf  INK 
$ ln -s ZEPTO_DIR/BGP-ION-ramdisk-for-CNL.elf ramdisk 
$ ln -s ../factory-default/CNS 
$ ln -s ../factory-default/uloader 

NOTE: your Zepto images must be readable from others, otherwise your job will fail. Please double check!!!

For ANL user, we provide a convenient script named mkprofile-ANL.sh which essentially does what mentioned in above but has some extra features. The following commend line is equivalent to the steps described in above.

$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME 

Invoking it with the -h option shows help message. Use -c if you actually need to copy images instead of making symbolic link. Use -cn, -ion or -rd if you have a custom named image.

$ ./mkprofile-ANL.sh -h 
Usage: ./mkprofile-ANL.sh [OPTIONS]   
 
Options: 
-h             : Show this message 
-c             : Copy images instead of making symbolic link 
-f             : Overwrite existing profile  
--profile=name : Specify profile name 
--cn=fn        : Compute Node Kernel Image   
--ion=fn       : Specify I/O Node Kernel Image       
--rd=fn        : Specify I/O Node Ramdisk Image 
--ls           : show files in profile 
--dryrun  

Once you have properly configured your Zepto kernel profile, you can boot Zepto kernel by specifying your kernel profile name via the -k cobalt option.

$ cqsub -k YOUR_PROFILE_NAME .... 

MMCS console

We explain how to assign and boot your own kernel images using Midplane Management Control System(MMCS). MMCS is the lowest control mechanism for BGP partition and installed on all BGP system. Here is the brief summary of MMCS.

• allocate, free or query of block(partition)
• status check
• assign boot images
• low level debug command

Due to its low level interface, it requires administrator level permission to use it. You also need to reserve a partition.

1. Assigned Zepto images to a partition that you will use
2. Start your job with the partition via mpi_run. Zepto kernel automatically starts.
3. Reset to the default kernel profile

Assign Zepto images to a BGP partition

Login to the service node and start MMCS

$ ssh sn         
sn $ ./mmcs.sh  

[mmcs.sh] 
#!/bin/sh 
 
export DB2HOME=/dbhome/bgpdb2c/sqllib 
DB2SRC=${DB2HOME}/db2profile 
[ -f "$DB2SRC" ] && . $DB2SRC 
 
cd /bgsys/drivers/ppcfloor/bin 
./mmcs_db_console 

Memorize that the current configuration. You need to revert the blockinfo to the original configuration after you have done using Zepto kernel.

console $ set_username YOUR_LOGIN_NAME 
console $ getblockinfo BGP_BLOCK_NAME 
OK 
boot info for block BGP_BLOCK_NAME: 
mloader: /bgsys/drivers/ppcfloor/boot/uloader 
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk 
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk 
status: F 

Assign Zepto images to a parition

console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,BGP_CN_LINUX_KERNEL_PATH /bgsys/drivers/ppcfloor\
/boot/cns,BGP_ION_LINUX_KERNEL_PATH,BGP_ION_LINUX_RAMDISK_PATH 
console $ quit 

Boot Zepto kernel

Once you have configured a partition with Zepto kernels correctly, Zepto kernels will be booted when you run a job on that partition(via mpirun for example)

fen $ mpirun -verbose 1 -partition BGP_BLOCK_NAME  -np 64 -timeout 600 -cwd `pwd` -exe ./a.out 

Restore to the original configuration(Don't forget!!!)

After you have done your work on Zepto kernel, you need to restore to the original configuration. Here is an example.

fen $ ssh sn 
sn $ ./mmcs.sh 
console $ set_username YOUR_LOGIN_NAME 
console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk /bgsys/drivers/\
ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk 
console $ quit