Kernel

We currently provide two Linux kernels because of GPFS support on ION.

• 2.6.19 based kernel : Zepto CN kernel
  • IBM V1R3 patch and zepto patch applied
  • 64KB pagesize and bigmemory region available
  • Device drivers for compute node devices such as DMA, lockbox, etc
  • Allow to run MPICH/DCMF code through Zepto Compute Binary(ZCB)
  • Can be used as enhanced ION kernel (no GPFS modules available)

• 2.6.16 based kernel : Zepto ION kernel
  • IBM V1R3 patch applied
  • Essentially same as IBM ION kernel. Support GPFS

Kernel directory structure

Kernel directory basically consists of three main directories ; prebuilt, config and tarball.

kernel
|-- prebuilt
|   |-- 2.6.16
|   |   `-- ION
|   `-- 2.6.19
|       |-- CN
|       `-- objs
|-- tarball
`-- config

The prebuilt directory contains prebuilt kernel images and kernel modules. While prebuilt ION kernel ELF file is found, no CN kernel ELF file is found here. This is because CN kernel ramdisk is embedded in kernel and we wanted to provide a way to replace the CNK ramdisk without invoking CN build process which requires untar'ing source code. Instead CN kernel objects are found in prebuilt directory and a CN kernel ELF is created from the objects and ramdisk every time when you invoke certain make target.

The tarball directory contains kernel tarballs separately for ION and CN Linux kernel. Technically, those tarball are a snopshot of the Zepto kernel git repository. The directory might contain a .patch file that contains difference between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git even for small modification. Associated git log file can be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot is created, so this information is used to create a patch file.

linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2

The config directory contains Linux kernel config. A config file is basically associated to a tarball by kernel version. i.e, bgp-cn-2.6.19.2-dot-config for linux-2.6.19.2-BGP-V1R3.tar.bz2 See also Makefile.

Build kernel

Makefile in the kernel directory has many options. Just type 'make' it will show you help message.

If you need to build (or rebuild) kernel from the tarball of kernel source code, use bgp-ion-linux-build or bgp-cn-linux-build target. By default it extracts ION or CN kernel tarball in a directory named work, apply a patch if any and start kernel build. Once kernel has successfully been built, kernel images (in both Zepto top directory and tmp directory) will be replaced with newly built images. ION kernel source code is extracted in work/linux-2.6.16.46-297-BGP-V1R3 and CN kernel source is in work/linux-2.6.19.2-BGP-V1R3. Here is an example of build, rebuilding kernel.

$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf

As mentioned above the kernel tarball is used as source code. If you specify GIT=1 to the make command, you can directly build from our Zepto kernel git tree. This is very useful for kernel development since you can keep track your modifications.

$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....

Note that repo/linux-2.6.19.2-BGP-V1R3 is a cloned git repository. See also BG-Linux page

Kernel config

Initially, config/bgp-cn-2.6.19.2-dot-config is applied CN Linux kernel build tree and config/bgp-ion-2.6.16.46-dot-config is applied to ION Linux kernel build tree. Technically when you type a kernel build target, associated kernel config file is copied to .config in kernel build directory.

bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig target invoke text based Linux kernel configuration menu.

$ make bgp-ion-linux-menuconfig
or
$ make bgp-cn-linux-menuconfig

Kernel (command line) Parameters

In usual server/desktop Linux environment, kernel parameters are passed via bootloader such as grub. However, BlueGene/P boot mechanism does not have such capability. Kernel parameters are very handy since we can skip kernel rebuild. So we have modified CN Linux kernel (2.6.19) to allow to load kernel parameter string embedded in kernel ELF image file.

You can add kernel parameters (or reset) to a kernel ELF file by a command line tool named zkparam.py, which is located in kernel directory. Here is the synopsis of the tool.

zkparam.py  KERNEL_ELF  [options]

If you omit options, the tool shows you the current kernel parameters.

$ ./kernel/zkparam.py  BGP-CN-zImage-with-initrd.elf  zepto_console_output=2
$ ./kernel/zkparam.py  BGP-CN-zImage-with-initrd.elf 
Current Kernel Parameters:
 zepto_console_output=2

Zepto kernel specific kernel parameters

• zepto_debug=INTEGER
  • Specify the zepto kernel debug level
  • 0 turns off all zepto debug msgs. higher number more detail
  • default=1
• flatmemsizeMB=INTEGER
  • Specify the size of flatmemory in MB.
  • Currently the granularity of memory size is limited to 256 MB
  • default=256 min=256 max=1792
• zepto_console_output=INTEGER
  • Specify the console output behavior.
  • 0 disables all console output from kernel.
  • 1 enables console output from one of nodes(personality rank=1)
  • 2 enables console output from all nodes

Log files,etc

Compute Node log

Debug messages from compute node (i.e, via printk) will appear in one of system log files. The system log file is recreated every time at system reset. You can find the location of the system log file by typing the following command.

$ ls -1 /bgsys/logs/BGP/sn*-mmcs_db_server*.log|tail -1
/bgsys/logs/BGP/sn1-mmcs_db_server-2009-0209-11:58:20.log

Please also take a look at a convenient script BGP/packages/tools/cn-log.sh

ION Node log

Debug message from ION node will appear in ION node log files in /bgsys/logs/BGP/, /bgsys/logs/BGP/R00-M0-N00-J00.log for example. Each ION has own log file. ION to CN ratio is 1:64 in ANL system. If your job is a 64 (physical) nodes job, you have one ION log file.

Please also take a look at a convenient script BGP/packages/tools/ion-log.sh

RAS events

RAS messages won't appear in the log files. A command line tool named bg-listevents shows you a record of RAS events. Type 'bg-listevents -h' for command line parameters.