Kernel

MPICH, DCMF, and SPI | Top | Ramdisk

---

Introduction

We currently provide two Linux kernels because of GPFS support on ION.

• 2.6.19-based kernel: ZeptoOS CN kernel
  • IBM V1R3 patch and ZeptoOS patch applied
  • 64 KB pagesize and big memory region available
  • Device drivers for compute node devices such as DMA, lockbox, etc
  • Allows to run MPICH/DCMF code through Zepto Compute Binary (ZCB)
  • Can be used as enhanced ION kernel

• 2.6.16-based kernel: ZeptoOS ION kernel
  • IBM V1R3 patch applied
  • Only minor changes compared to the IBM ION kernel.

We focus our development efforts on the 2.6.19-based kernel. It is meant primarily for the compute nodes, but can also be used on the I/O nodes. The problem is that GPFS does not work with this kernel, so we also provide the 2.6.16-based kernel which works with GPFS.

Kernel directory structure

The kernel directory basically consists of three main subdirectories: prebuilt, config, and tarball.

kernel
|-- prebuilt
|   |-- 2.6.16
|   |   `-- ION
|   `-- 2.6.19
|       |-- CN
|       `-- objs
|-- tarball
`-- config

The prebuilt directory contains prebuilt kernel images and modules. While a complete prebuilt ION kernel ELF file is provided, for the CN kernel we provide intermediate object files instead. This is because we embed the CN ramdisk in the CN kernel image when building ZeptoOS, and this process requires the object files.

The tarball directory contains kernel tarballs separately for the ION and the CN Linux kernel. Technically, those tarballs are a snapshot of the ZeptoOS kernel git repository. The directory might contain a .patch file that contains the differences between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git for small modifications. Associated git log file can also be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot was created, so this information is used to create a patch file.

Here is a list of files for the CN kernel:

linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2

The config directory contains Linux kernel configs. In case of the 2.6.19 kernel, we provide separate config files for the compute node and the I/O node.

Building a kernel

Makefile in the kernel directory has many options. Just type make and it will print out a help message.

If one needs to build (or rebuild) a kernel from the source tarball, use bgp-ion-linux-build or bgp-cn-linux-build target. By default, it extracts ION or CN kernel tarball in a directory named work, applies a patch if any and starts the kernel build. Once the kernel has successfully been built, kernel images (in both ZeptoOS top-level directory and the tmp directory) will be replaced with newly built images. The ION kernel source code is extracted into work/linux-2.6.16.46-297-BGP-V1R3 and the CN kernel source into work/linux-2.6.19.2-BGP-V1R3.

Here is an example of building and rebuilding the CN kernel:

$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf

Building a kernel from the ZeptoOS kernel git repository

As mentioned earlier, the kernel tarball is used as the source by default. If instead one passes GIT=1 to make, one can build directly from the ZeptoOS kernel git tree. This is very useful for kernel development since it makes it easier to keep track of local modifications.

$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....

This will create repo/linux-2.6.19.2-BGP-V1R3, which is a git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you cannot push your modifications to it. Instead, please post any patches to the ZeptoOS mailing list instead.

See also the BG-Linux page for the details on our kernel git repository.

Kernel config

When one invokes make with a kernel build target for the first time, the associated kernel config file is copied to .config in the kernel build directory. config/bgp-cn-2.6.19.2-dot-config is applied to the CN Linux kernel build tree, and config/bgp-ion-2.6.16.46-dot-config is applied to the ION Linux kernel build tree.

Here is the location of the kernel config file:

• Regular build
  • work/build-2.6.19.2-BGP-V1R3/.config
  • work/build-2.6.16.46-297-BGP-V1R3/.config
• GIT build
  • repo/build-2.6.19.2-BGP-V1R3/.config
  • repo/build-2.6.16.46-297-BGP-V1R3/.config

Please note that the kernel config file is copied only once, until you do a distclean or remove the files manually.

The bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig make targets invoke text-based Linux kernel configuration menus:

$ make bgp-ion-linux-menuconfig
$ make bgp-cn-linux-menuconfig

For GIT build:

$ make GIT=1 bgp-ion-linux-menuconfig
$ make GIT=1 bgp-cn-linux-menuconfig

These menu targets never update the default kernel config files from the config directory. If you want to apply a new config permanently, please copy it to the config directory by hand:

$ cp work/build-2.6.19.2-BGP-V1R3/.config  config/bgp-cn-2.6.19.2-dot-config

Kernel (command line) arguments

In common server/desktop Linux environments, kernel parameters can be passed via bootloader such as grub. However, BlueGene/P boot mechanism does not have such capability. Kernel parameters are very handy, since we can skip kernel rebuild. So we have modified the CN Linux kernel (2.6.19) to allow to load kernel parameter string embedded in kernel ELF image file.

You can add kernel parameters (or reset) to a kernel ELF file by a command line tool named zkparam.py, which is located in kernel directory. Here is the synopsis of the tool.

zkparam.py  KERNEL_ELF  [options]

If you omit options, the tool shows you the current kernel parameters.

$ ./kernel/zkparam.py  BGP-CN-zImage-with-initrd.elf  zepto_console_output=2
$ ./kernel/zkparam.py  BGP-CN-zImage-with-initrd.elf 
Current Kernel Parameters:
 zepto_console_output=2

Zepto kernel specific kernel parameters

• zepto_debug=INTEGER
  • Specify the zepto kernel debug level
  • 0 turns off all zepto debug msgs. higher number more detail
  • default=1
• flatmemsizeMB=INTEGER
  • Specify the size of flatmemory in MB.
  • Currently the granularity of memory size is limited to 256 MB
  • default=256 min=256 max=1792
• zepto_console_output=INTEGER
  • Specify the console output behavior.
  • 0 disables all console output from kernel.
  • 1 enables console output from one of nodes(personality rank=1)
  • 2 enables console output from all nodes

Log files, etc

The compute node and I/O node logfile have been discussed extensively in Testing.

In addition to regular console logs, the kernels can also generate RAS message, which will not appear in the log files. A command line tool named bg-listevents shows you a record of RAS events. Type bg-listevents -h for command line arguments.

---

MPICH, DCMF, and SPI | Top | Ramdisk