Difference between revisions of "Kernel"

From ZeptoOS
Jump to navigationJump to search
 
(25 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
We currently provide two Linux kernels because of GPFS support on ION.
+
[[MPICH, DCMF, and SPI]] | [[ZeptoOS_Documentation|Top]] | [[Ramdisk]]
 +
----
  
−
* 2.6.19 based kernel : Zepto CN kernel
+
==Introduction==
−
** IBM V1R3 patch and zepto patch applied
+
 
−
** 64KB pagesize and bigmemory region available
+
We currently provide two Linux kernels:
 +
 
 +
* 2.6.19-based kernel: ZeptoOS CN kernel
 +
** IBM V1R3 patch and ZeptoOS patch applied
 +
** 64 KB page size and big memory region available
 
** Device drivers for compute node devices such as DMA, lockbox, etc
 
** Device drivers for compute node devices such as DMA, lockbox, etc
−
** Allow to run MPICH/DCMF code through Zepto Compute Binary(ZCB)
+
** Allows to run MPICH/DCMF code through Zepto Compute Binary (ZCB)
−
** Can be used as enhanced ION kernel (no GPFS modules available)
+
** Can be used as enhanced ION kernel
  
−
* 2.6.16 based kernel : Zepto ION kernel
+
* 2.6.16-based kernel: ZeptoOS ION kernel
 
** IBM V1R3 patch applied
 
** IBM V1R3 patch applied
−
** Essentially same as IBM ION kernel. Support GPFS
+
** Only minor changes compared to the IBM ION kernel.
 +
 
 +
We focus our development efforts on the 2.6.19-based kernel.  It is meant primarily for the compute nodes, but can also be used on the I/O nodes.  The problem is that GPFS does not work with this kernel, so we also provide the 2.6.16-based kernel which GPFS does work with.
  
 
==Kernel directory structure==
 
==Kernel directory structure==
  
−
Kernel directory basically consists of three main directories ; prebuilt, config and tarball.
+
The <tt>kernel</tt> directory consists of three main subdirectories: <tt>prebuilt</tt>, <tt>config</tt>, and <tt>tarball</tt>.
  
 
<pre>
 
<pre>
−
|-- config
+
kernel
 
|-- prebuilt
 
|-- prebuilt
 
|  |-- 2.6.16
 
|  |-- 2.6.16
Line 24: Line 31:
 
|      |-- CN
 
|      |-- CN
 
|      `-- objs
 
|      `-- objs
−
`-- tarball
+
|-- tarball
 +
`-- config
 
</pre>
 
</pre>
  
−
The prebuilt directory contains prebuilt kernel images and kernel modules. While prebuilt ION kernel  
+
The <tt>prebuilt</tt> directory contains prebuilt kernel images and modules. While a complete prebuilt ION kernel ELF file is provided, for the CN kernel we provide intermediate object files instead. This is because we embed the CN ramdisk in the CN kernel image when building ZeptoOS, and this process requires the object files.
−
ELF file is found, no CN kernel ELF file is found here. This is because CN kernel ramdisk is embedded in kernel  
+
 
−
and we wanted to provide a way to replace the CNK ramdisk without invoking CN build process which requires  
+
The <tt>tarball</tt> directory contains kernel tarballs separately for the ION and the CN Linux kernel. Technically, those tarballs are snapshots of the ZeptoOS kernel git repository. The directory might contain a <tt>.patch</tt> file that contains the differences between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git for small modifications. Associated git log files can also be found in this directory. A <tt>.SNAPSHOT_HEAD</tt> file indicates the git revision at the time when a snapshot was created, so this information is used to create a patch file.
−
untar'ing source code. Instead CN kernel objects are found in prebuilt directory and
+
 
−
a CN kernel ELF is created from the objects and ramdisk every time when you invoke certain make target.
+
As an example, here is a list of files for the CN kernel:
  
−
The tarball directory contains kernel tarballs separately for ION and CN Linux kernel. Technically, those tarball are a snopshot of
 
−
the Zepto kernel git repository. The directory might contain a .patch file that contains difference between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git even for small modification.
 
−
Associated git log file can be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot is created, so this information is used to create a patch file.
 
 
<pre>
 
<pre>
 
linux-2.6.19.2-BGP-V1R3.git.log
 
linux-2.6.19.2-BGP-V1R3.git.log
Line 43: Line 48:
 
</pre>
 
</pre>
  
−
The config file contains Linux kernel config.   
+
The <tt>config</tt> directory contains Linux kernel configs.  In case of the 2.6.19 kernel, we provide separate config files for the compute node and the I/O node.
 +
 
 +
==Building a kernel==
 +
 
 +
The <tt>Makefile</tt> in the <tt>kernel</tt> directory has many targets. Just type <tt>make</tt> and it will print out a help message.
 +
 
 +
If one needs to build (or rebuild) a kernel from a source tarball, use <tt>bgp-ion-linux-build</tt> or <tt>bgp-cn-linux-build</tt> targets. By default, it extracts ION or CN kernel tarball in a directory named <tt>work</tt>, applies a patch if any and starts the kernel build. Once the kernel has successfully been built, kernel images (in both ZeptoOS top-level directory and the <tt>tmp</tt> directory) will be replaced with newly built ones. The ION kernel source code is extracted into <tt>work/linux-2.6.16.46-297-BGP-V1R3</tt> and the CN kernel source into <tt>work/linux-2.6.19.2-BGP-V1R3</tt>.
 +
 
 +
Here is an example of building and rebuilding the CN kernel:
 +
 
 +
<pre>
 +
$ cd kernel
 +
$ make bgp-cn-linux-build
 +
....
 +
$ ls -al ../BGP-CN-zImage-with-initrd.elf
 +
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
 +
$ make bgp-cn-linux-build
 +
....
 +
$ ls -al ../BGP-CN-zImage-with-initrd.elf
 +
</pre>
 +
 
 +
===Building a kernel from the ZeptoOS kernel git repository===
 +
 
 +
As mentioned earlier, kernel tarballs are used as the source by default.  If instead one passes <tt>GIT=1</tt> to <tt>make</tt>, one can build directly from the ZeptoOS kernel git tree. This is very useful for kernel development since it makes it easier to keep track of local modifications.
 +
 
 +
<pre>
 +
$ cd kernel
 +
$ make GIT=1 bgp-cn-linux-build
 +
....
 +
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
 +
$ make GIT=1 bgp-cn-linux-build
 +
....
 +
</pre>
 +
 
 +
This will create <tt>repo/linux-2.6.19.2-BGP-V1R3</tt>, which is a git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you cannot push your modifications to it. Instead, please post any patches to the [mailto:[email protected] ZeptoOS developers mailing list].
 +
 
 +
See also the [http://bg-linux.anl-external.org/wiki/index.php/Main_Page BG-Linux page] for the details on our kernel git repository.
 +
 
 +
===Kernel config===
 +
 
 +
When one invokes <tt>make</tt> with a kernel build target for the first time, the associated kernel config file is copied to <tt>.config</tt> in the kernel build directory.  <tt>config/bgp-cn-2.6.19.2-dot-config</tt> is applied to the CN Linux kernel build tree, and <tt>config/bgp-ion-2.6.16.46-dot-config</tt> is applied to the ION Linux kernel build tree.
 +
 
 +
Here are the locations of the kernel config files:
 +
* Regular build
 +
** work/build-2.6.19.2-BGP-V1R3/.config
 +
** work/build-2.6.16.46-297-BGP-V1R3/.config
 +
* GIT build
 +
** repo/build-2.6.19.2-BGP-V1R3/.config
 +
** repo/build-2.6.16.46-297-BGP-V1R3/.config
 +
 
 +
Please note that the kernel config file is copied only once, until one does a <tt>distclean</tt> or removes the files manually.
 +
 
 +
The <tt>bgp-cn-linux-menuconfig</tt> and <tt>bgp-ion-linux-menuconfig</tt> <tt>make</tt> targets invoke text-based Linux kernel configuration menus:
 +
 
 +
<pre>
 +
$ make bgp-ion-linux-menuconfig
 +
$ make bgp-cn-linux-menuconfig
 +
</pre>
 +
 
 +
For GIT build:
 +
<pre>
 +
$ make GIT=1 bgp-ion-linux-menuconfig
 +
$ make GIT=1 bgp-cn-linux-menuconfig
 +
</pre>
 +
 
 +
These menu targets never update the default kernel config files from the <tt>config</tt> directory.  If one wants to apply a new config permanently, please copy it to the <tt>config</tt> directory by hand:
 +
<pre>
 +
$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config
 +
</pre>
 +
 
 +
==Kernel (command line) parameters==
 +
 
 +
In common server/desktop Linux environments, kernel parameters can be passed via a bootloader such as grub. However, Blue Gene/P boot mechanism does not provide such capability, so we have modified the CN Linux kernel (2.6.19) to use a kernel parameter string embedded in kernel ELF image file itself.
 +
 
 +
One can (re)set the kernel parameters in a kernel ELF file using a command line tool <tt>zkparam.py</tt>, located in the <tt>bin</tt> subdirectory of the ZeptoOS installation directory.  Here is the synopsis of the tool:
 +
 
 +
<pre>
 +
zkparam.py <kernel_image> [options]
 +
</pre>
 +
 
 +
If options are omitted, the tool shows the current kernel parameters:
 +
 
 +
<pre>
 +
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
 +
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf
 +
Current Kernel Parameters:
 +
zepto_console_output=2
 +
</pre>
 +
 
 +
===ZeptoOS-pecific kernel parameters===
 +
 
 +
* '''zepto_debug'''=<integer>
 +
** Specifies the ZeptoOS kernel debug level.
 +
** The higher the number, the more messages are generated.
 +
** <tt>0</tt> turns off all debug messages.
 +
** default=1
 +
* '''flatmemsizeMB'''=<integer>
 +
** Specifies the size of big memory in MB.
 +
** Currently the granularity of memory size is 256&nbsp;MB.
 +
** default=256  min=256  max=1792
 +
* '''zepto_console_output'''=<integer>
 +
** Specifies the console output behavior.
 +
** 0 disables console output from all compute nodes.
 +
** 1 enables console output from the first compute node ([[FAQ#Torus rank|torus rank]] 0).
 +
** 2 enables console output from all compute nodes.
 +
 
 +
==Log files, etc==
 +
<!--
 +
===Compute Node log===
 +
 
 +
Debug messages from compute node (i.e, via printk) will appear in one of system log files.
 +
The system log file is recreated every time at system reset. You can find the location of the system log file
 +
by typing the following command.
 +
 
 +
<pre>
 +
$ ls -1 /bgsys/logs/BGP/sn*-mmcs_db_server*.log|tail -1
 +
/bgsys/logs/BGP/sn1-mmcs_db_server-2009-0209-11:58:20.log
 +
</pre>
 +
 
 +
Please also take a look at a convenient script BGP/packages/tools/cn-log.sh
 +
 
 +
===ION Node log===
 +
 
 +
Debug message from ION node will appear in ION node log files in /bgsys/logs/BGP/,
 +
/bgsys/logs/BGP/R00-M0-N00-J00.log for example.
 +
Each ION has own log file. ION to CN ratio is 1:64 in ANL system. If your job is a 64 (physical) nodes job,
 +
you have one ION log file.
 +
 
 +
Please also take a look at a convenient script BGP/packages/tools/ion-log.sh
 +
 
 +
===RAS events===
 +
-->
 +
 
 +
The compute node and I/O node logfiles have been discussed extensively in [[Testing#Log files|Testing]].
 +
 
 +
In addition to regular console logs, the kernels can also generate RAS messages, which will not appear in the log files; they are instead stored in a database on the service node. At Argonne we have a custom command line tool named <tt>bg-listevents</tt> that shows a record of RAS events (type <tt>bg-listevents -h</tt> for command line arguments).
  
−
==BUILD==
+
----
 +
[[MPICH, DCMF, and SPI]] | [[ZeptoOS_Documentation|Top]] | [[Ramdisk]]

Latest revision as of 13:58, 8 May 2009

MPICH, DCMF, and SPI | Top | Ramdisk

Introduction

We currently provide two Linux kernels:

  • 2.6.19-based kernel: ZeptoOS CN kernel
    • IBM V1R3 patch and ZeptoOS patch applied
    • 64 KB page size and big memory region available
    • Device drivers for compute node devices such as DMA, lockbox, etc
    • Allows to run MPICH/DCMF code through Zepto Compute Binary (ZCB)
    • Can be used as enhanced ION kernel
  • 2.6.16-based kernel: ZeptoOS ION kernel
    • IBM V1R3 patch applied
    • Only minor changes compared to the IBM ION kernel.

We focus our development efforts on the 2.6.19-based kernel. It is meant primarily for the compute nodes, but can also be used on the I/O nodes. The problem is that GPFS does not work with this kernel, so we also provide the 2.6.16-based kernel which GPFS does work with.

Kernel directory structure

The kernel directory consists of three main subdirectories: prebuilt, config, and tarball. 

kernel
|-- prebuilt
|   |-- 2.6.16
|   |   `-- ION
|   `-- 2.6.19
|       |-- CN
|       `-- objs
|-- tarball
`-- config

The prebuilt directory contains prebuilt kernel images and modules. While a complete prebuilt ION kernel ELF file is provided, for the CN kernel we provide intermediate object files instead. This is because we embed the CN ramdisk in the CN kernel image when building ZeptoOS, and this process requires the object files.

The tarball directory contains kernel tarballs separately for the ION and the CN Linux kernel. Technically, those tarballs are snapshots of the ZeptoOS kernel git repository. The directory might contain a .patch file that contains the differences between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git for small modifications. Associated git log files can also be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot was created, so this information is used to create a patch file.

As an example, here is a list of files for the CN kernel: 

linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2

The config directory contains Linux kernel configs. In case of the 2.6.19 kernel, we provide separate config files for the compute node and the I/O node.

Building a kernel

The Makefile in the kernel directory has many targets. Just type make and it will print out a help message.

If one needs to build (or rebuild) a kernel from a source tarball, use bgp-ion-linux-build or bgp-cn-linux-build targets. By default, it extracts ION or CN kernel tarball in a directory named work, applies a patch if any and starts the kernel build. Once the kernel has successfully been built, kernel images (in both ZeptoOS top-level directory and the tmp directory) will be replaced with newly built ones. The ION kernel source code is extracted into work/linux-2.6.16.46-297-BGP-V1R3 and the CN kernel source into work/linux-2.6.19.2-BGP-V1R3.

Here is an example of building and rebuilding the CN kernel: 

$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf

Building a kernel from the ZeptoOS kernel git repository

As mentioned earlier, kernel tarballs are used as the source by default. If instead one passes GIT=1 to make, one can build directly from the ZeptoOS kernel git tree. This is very useful for kernel development since it makes it easier to keep track of local modifications. 

$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....

This will create repo/linux-2.6.19.2-BGP-V1R3, which is a git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you cannot push your modifications to it. Instead, please post any patches to the ZeptoOS developers mailing list.

See also the BG-Linux page for the details on our kernel git repository.

Kernel config

When one invokes make with a kernel build target for the first time, the associated kernel config file is copied to .config in the kernel build directory. config/bgp-cn-2.6.19.2-dot-config is applied to the CN Linux kernel build tree, and config/bgp-ion-2.6.16.46-dot-config is applied to the ION Linux kernel build tree.

Here are the locations of the kernel config files:

  • Regular build
    • work/build-2.6.19.2-BGP-V1R3/.config
    • work/build-2.6.16.46-297-BGP-V1R3/.config
  • GIT build
    • repo/build-2.6.19.2-BGP-V1R3/.config
    • repo/build-2.6.16.46-297-BGP-V1R3/.config

Please note that the kernel config file is copied only once, until one does a distclean or removes the files manually.

The bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig make targets invoke text-based Linux kernel configuration menus: 

$ make bgp-ion-linux-menuconfig
$ make bgp-cn-linux-menuconfig

For GIT build: 

$ make GIT=1 bgp-ion-linux-menuconfig
$ make GIT=1 bgp-cn-linux-menuconfig

These menu targets never update the default kernel config files from the config directory. If one wants to apply a new config permanently, please copy it to the config directory by hand: 

$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config

Kernel (command line) parameters

In common server/desktop Linux environments, kernel parameters can be passed via a bootloader such as grub. However, Blue Gene/P boot mechanism does not provide such capability, so we have modified the CN Linux kernel (2.6.19) to use a kernel parameter string embedded in kernel ELF image file itself.

One can (re)set the kernel parameters in a kernel ELF file using a command line tool zkparam.py, located in the bin subdirectory of the ZeptoOS installation directory. Here is the synopsis of the tool: 

zkparam.py <kernel_image> [options]

If options are omitted, the tool shows the current kernel parameters: 

$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf
Current Kernel Parameters:
 zepto_console_output=2

ZeptoOS-pecific kernel parameters

  • zepto_debug=<integer>
    • Specifies the ZeptoOS kernel debug level.
    • The higher the number, the more messages are generated.
    • 0 turns off all debug messages.
    • default=1
  • flatmemsizeMB=<integer>
    • Specifies the size of big memory in MB.
    • Currently the granularity of memory size is 256 MB.
    • default=256 min=256 max=1792
  • zepto_console_output=<integer>
    • Specifies the console output behavior.
    • 0 disables console output from all compute nodes.
    • 1 enables console output from the first compute node (torus rank 0).
    • 2 enables console output from all compute nodes.

Log files, etc

The compute node and I/O node logfiles have been discussed extensively in Testing.

In addition to regular console logs, the kernels can also generate RAS messages, which will not appear in the log files; they are instead stored in a database on the service node. At Argonne we have a custom command line tool named bg-listevents that shows a record of RAS events (type bg-listevents -h for command line arguments).

MPICH, DCMF, and SPI | Top | Ramdisk

Retrieved from "http:/// /zeptoos/index.php?title=Kernel&oldid=601"