Kernel

MPICH, DCMF, and SPI | Top | Ramdisk

---

Introduction

We currently provide two Linux kernels because of GPFS support on ION.

• 2.6.19 based kernel : Zepto CN kernel
  • IBM V1R3 patch and zepto patch applied
  • 64KB pagesize and bigmemory region available
  • Device drivers for compute node devices such as DMA, lockbox, etc
  • Allow to run MPICH/DCMF code through Zepto Compute Binary(ZCB)
  • Can be used as enhanced ION kernel (no GPFS modules available)

• 2.6.16 based kernel : Zepto ION kernel
  • IBM V1R3 patch applied
  • Essentially same as IBM ION kernel. Support GPFS

Kernel directory structure

Kernel directory basically consists of three main directories ; prebuilt, config and tarball.

kernel
|-- prebuilt
|   |-- 2.6.16
|   |   `-- ION
|   `-- 2.6.19
|       |-- CN
|       `-- objs
|-- tarball
`-- config

The prebuilt directory contains prebuilt kernel images and kernel modules. While prebuilt ION kernel ELF file is found, no CN kernel ELF file is found here. This is because CN kernel ramdisk is embedded in kernel and we wanted to provide a way to replace the CNK ramdisk without invoking CN build process which requires untar'ing source code. Instead CN kernel objects are found in prebuilt directory and a CN kernel ELF is created from the objects and ramdisk every time when you invoke certain make target.

The tarball directory contains kernel tarballs separately for ION and CN Linux kernel. Technically, those tarball are a snopshot of the Zepto kernel git repository. The directory might contain a .patch file that contains difference between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git even for small modification. Associated git log file can be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot is created, so this information is used to create a patch file.

linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2

The config directory contains Linux kernel config. A config file is basically associated to a tarball by kernel version. i.e, bgp-cn-2.6.19.2-dot-config for linux-2.6.19.2-BGP-V1R3.tar.bz2 See also Makefile.

Build kernel

Makefile in the kernel directory has many options. Just type 'make' it will show you help message.

If you need to build (or rebuild) kernel from the tarball of kernel source code, use bgp-ion-linux-build or bgp-cn-linux-build target. By default it extracts ION or CN kernel tarball in a directory named work, apply a patch if any and start kernel build. Once kernel has successfully been built, kernel images (in both Zepto top directory and tmp directory) will be replaced with newly built images. ION kernel source code is extracted in work/linux-2.6.16.46-297-BGP-V1R3 and CN kernel source is in work/linux-2.6.19.2-BGP-V1R3. Here is an example of build, rebuilding kernel.

$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf

Build kernel from Zepto kernel git repo

As mentioned above the kernel tarball is used as source code by default. If you specify GIT=1 to the make command, you can directly build from our Zepto kernel git tree. This is very useful for kernel development since you can keep track your modifications.

$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....

Note that repo/linux-2.6.19.2-BGP-V1R3 is a cloned git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you can't push your modifications to the http repo. You can post your git patch to ZeptoOS mailing list first.

See also BG-Linux page for our kernel git repo details.

Kernel config

Initially, config/bgp-cn-2.6.19.2-dot-config is applied CN Linux kernel build tree and config/bgp-ion-2.6.16.46-dot-config is applied to ION Linux kernel build tree. Technically when you type a kernel build target the first time, associated kernel config file is copied to .config in kernel build directory.

Here is the location of kernel config file.

• Regular build
  • work/build-2.6.19.2-BGP-V1R3/.config
  • work/build-2.6.16.46-297-BGP-V1R3/.config
• GIT build
  • repo/build-2.6.19.2-BGP-V1R3/.config
  • repo/build-2.6.16.46-297-BGP-V1R3/.config

Please note that the kernel config file is copied only once until you do distclean or remove the files manually.

bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig target invoke text based Linux kernel configuration menu.

$ make bgp-ion-linux-menuconfig
$ make bgp-cn-linux-menuconfig

For GIT build,

$ make GIT=1 bgp-ion-linux-menuconfig
$ make GIT=1 bgp-cn-linux-menuconfig

Those menu target never update default kernel config file in the config directory. If you want to apply new config permanently, please copy it to the config directory by hand. For example,

$ cp work/build-2.6.19.2-BGP-V1R3/.config  config/bgp-cn-2.6.19.2-dot-config

Kernel (command line) Parameters

In usual server/desktop Linux environment, kernel parameters are passed via bootloader such as grub. However, BlueGene/P boot mechanism does not have such capability. Kernel parameters are very handy since we can skip kernel rebuild. So we have modified CN Linux kernel (2.6.19) to allow to load kernel parameter string embedded in kernel ELF image file.

You can add kernel parameters (or reset) to a kernel ELF file by a command line tool named zkparam.py, which is located in kernel directory. Here is the synopsis of the tool.

zkparam.py  KERNEL_ELF  [options]

If you omit options, the tool shows you the current kernel parameters.

$ ./kernel/zkparam.py  BGP-CN-zImage-with-initrd.elf  zepto_console_output=2
$ ./kernel/zkparam.py  BGP-CN-zImage-with-initrd.elf 
Current Kernel Parameters:
 zepto_console_output=2

Zepto kernel specific kernel parameters

• zepto_debug=INTEGER
  • Specify the zepto kernel debug level
  • 0 turns off all zepto debug msgs. higher number more detail
  • default=1
• flatmemsizeMB=INTEGER
  • Specify the size of flatmemory in MB.
  • Currently the granularity of memory size is limited to 256 MB
  • default=256 min=256 max=1792
• zepto_console_output=INTEGER
  • Specify the console output behavior.
  • 0 disables all console output from kernel.
  • 1 enables console output from one of nodes(personality rank=1)
  • 2 enables console output from all nodes

Log files,etc

Compute Node log

Debug messages from compute node (i.e, via printk) will appear in one of system log files. The system log file is recreated every time at system reset. You can find the location of the system log file by typing the following command.

$ ls -1 /bgsys/logs/BGP/sn*-mmcs_db_server*.log|tail -1
/bgsys/logs/BGP/sn1-mmcs_db_server-2009-0209-11:58:20.log

Please also take a look at a convenient script BGP/packages/tools/cn-log.sh

ION Node log

Debug message from ION node will appear in ION node log files in /bgsys/logs/BGP/, /bgsys/logs/BGP/R00-M0-N00-J00.log for example. Each ION has own log file. ION to CN ratio is 1:64 in ANL system. If your job is a 64 (physical) nodes job, you have one ION log file.

Please also take a look at a convenient script BGP/packages/tools/ion-log.sh

RAS events

RAS messages won't appear in the log files. A command line tool named bg-listevents shows you a record of RAS events. Type 'bg-listevents -h' for command line parameters.

---

MPICH, DCMF, and SPI | Top | Ramdisk