Kernel

MPICH, DCMF, and SPI | Top | Ramdisk

---

Introduction

We currently provide two Linux kernels:

• 2.6.19-based kernel: ZeptoOS CN kernel
  • IBM V1R3 patch and ZeptoOS patch applied
  • 64 KB page size and big memory region available
  • Device drivers for compute node devices such as DMA, lockbox, etc
  • Allows to run MPICH/DCMF code through Zepto Compute Binary (ZCB)
  • Can be used as enhanced ION kernel

• 2.6.16-based kernel: ZeptoOS ION kernel
  • IBM V1R3 patch applied
  • Only minor changes compared to the IBM ION kernel.

We focus our development efforts on the 2.6.19-based kernel. It is meant primarily for the compute nodes, but can also be used on the I/O nodes. The problem is that GPFS does not work with this kernel, so we also provide the 2.6.16-based kernel which GPFS does work with.

Kernel directory structure

The kernel directory consists of three main subdirectories: prebuilt, config, and tarball.

kernel
|-- prebuilt
|   |-- 2.6.16
|   |   `-- ION
|   `-- 2.6.19
|       |-- CN
|       `-- objs
|-- tarball
`-- config

The prebuilt directory contains prebuilt kernel images and modules. While a complete prebuilt ION kernel ELF file is provided, for the CN kernel we provide intermediate object files instead. This is because we embed the CN ramdisk in the CN kernel image when building ZeptoOS, and this process requires the object files.

The tarball directory contains kernel tarballs separately for the ION and the CN Linux kernel. Technically, those tarballs are snapshots of the ZeptoOS kernel git repository. The directory might contain a .patch file that contains the differences between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git for small modifications. Associated git log files can also be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot was created, so this information is used to create a patch file.

As an example, here is a list of files for the CN kernel:

linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2

The config directory contains Linux kernel configs. In case of the 2.6.19 kernel, we provide separate config files for the compute node and the I/O node.

Building a kernel

The Makefile in the kernel directory has many targets. Just type make and it will print out a help message.

If one needs to build (or rebuild) a kernel from a source tarball, use bgp-ion-linux-build or bgp-cn-linux-build targets. By default, it extracts ION or CN kernel tarball in a directory named work, applies a patch if any and starts the kernel build. Once the kernel has successfully been built, kernel images (in both ZeptoOS top-level directory and the tmp directory) will be replaced with newly built ones. The ION kernel source code is extracted into work/linux-2.6.16.46-297-BGP-V1R3 and the CN kernel source into work/linux-2.6.19.2-BGP-V1R3.

Here is an example of building and rebuilding the CN kernel:

$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf

Building a kernel from the ZeptoOS kernel git repository

As mentioned earlier, kernel tarballs are used as the source by default. If instead one passes GIT=1 to make, one can build directly from the ZeptoOS kernel git tree. This is very useful for kernel development since it makes it easier to keep track of local modifications.

$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....

This will create repo/linux-2.6.19.2-BGP-V1R3, which is a git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you cannot push your modifications to it. Instead, please post any patches to the ZeptoOS developers mailing list.

See also the BG-Linux page for the details on our kernel git repository.

Kernel config

When one invokes make with a kernel build target for the first time, the associated kernel config file is copied to .config in the kernel build directory. config/bgp-cn-2.6.19.2-dot-config is applied to the CN Linux kernel build tree, and config/bgp-ion-2.6.16.46-dot-config is applied to the ION Linux kernel build tree.

Here are the locations of the kernel config files:

• Regular build
  • work/build-2.6.19.2-BGP-V1R3/.config
  • work/build-2.6.16.46-297-BGP-V1R3/.config
• GIT build
  • repo/build-2.6.19.2-BGP-V1R3/.config
  • repo/build-2.6.16.46-297-BGP-V1R3/.config

Please note that the kernel config file is copied only once, until one does a distclean or removes the files manually.

The bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig make targets invoke text-based Linux kernel configuration menus:

$ make bgp-ion-linux-menuconfig
$ make bgp-cn-linux-menuconfig

For GIT build:

$ make GIT=1 bgp-ion-linux-menuconfig
$ make GIT=1 bgp-cn-linux-menuconfig

These menu targets never update the default kernel config files from the config directory. If one wants to apply a new config permanently, please copy it to the config directory by hand:

$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config

Kernel (command line) parameters

In common server/desktop Linux environments, kernel parameters can be passed via a bootloader such as grub. However, Blue Gene/P boot mechanism does not provide such capability, so we have modified the CN Linux kernel (2.6.19) to use a kernel parameter string embedded in kernel ELF image file itself.

One can (re)set the kernel parameters in a kernel ELF file using a command line tool zkparam.py, located in the bin subdirectory of the ZeptoOS installation directory. Here is the synopsis of the tool:

zkparam.py <kernel_image> [options]

If options are omitted, the tool shows the current kernel parameters:

$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf
Current Kernel Parameters:
 zepto_console_output=2

ZeptoOS-pecific kernel parameters

• zepto_debug=<integer>
  • Specifies the ZeptoOS kernel debug level.
  • The higher the number, the more messages are generated.
  • 0 turns off all debug messages.
  • default=1
• flatmemsizeMB=<integer>
  • Specifies the size of big memory in MB.
  • Currently the granularity of memory size is 256 MB.
  • default=256 min=256 max=1792
• zepto_console_output=<integer>
  • Specifies the console output behavior.
  • 0 disables console output from all compute nodes.
  • 1 enables console output from the first compute node (torus rank 0).
  • 2 enables console output from all compute nodes.

Log files, etc

The compute node and I/O node logfiles have been discussed extensively in Testing.

In addition to regular console logs, the kernels can also generate RAS messages, which will not appear in the log files; they are instead stored in a database on the service node. At Argonne we have a custom command line tool named bg-listevents that shows a record of RAS events (type bg-listevents -h for command line arguments).

---

MPICH, DCMF, and SPI | Top | Ramdisk