Difference between revisions of "Kernel"

From ZeptoOS
Jump to navigationJump to search
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
We currently provide two Linux kernels because of GPFS support on ION.
+
[[MPICH, DCMF, and SPI]] | [[ZeptoOS_Documentation|Top]] | [[Ramdisk]]
 +
----
  
−
* 2.6.19 based kernel : Zepto CN kernel
+
==Introduction==
−
** IBM V1R3 patch and zepto patch applied
+
 
−
** 64KB pagesize and bigmemory region available
+
We currently provide two Linux kernels:
 +
 
 +
* 2.6.19-based kernel: ZeptoOS CN kernel
 +
** IBM V1R3 patch and ZeptoOS patch applied
 +
** 64 KB page size and big memory region available
 
** Device drivers for compute node devices such as DMA, lockbox, etc
 
** Device drivers for compute node devices such as DMA, lockbox, etc
−
** Allow to run MPICH/DCMF code through Zepto Compute Binary(ZCB)
+
** Allows to run MPICH/DCMF code through Zepto Compute Binary (ZCB)
−
** Can be used as enhanced ION kernel (no GPFS modules available)
+
** Can be used as enhanced ION kernel
  
−
* 2.6.16 based kernel : Zepto ION kernel
+
* 2.6.16-based kernel: ZeptoOS ION kernel
 
** IBM V1R3 patch applied
 
** IBM V1R3 patch applied
−
** Essentially same as IBM ION kernel. Support GPFS
+
** Only minor changes compared to the IBM ION kernel.
 +
 
 +
We focus our development efforts on the 2.6.19-based kernel.  It is meant primarily for the compute nodes, but can also be used on the I/O nodes.  The problem is that GPFS does not work with this kernel, so we also provide the 2.6.16-based kernel which GPFS does work with.
  
 
==Kernel directory structure==
 
==Kernel directory structure==
  
−
Kernel directory basically consists of three main directories ; prebuilt, config and tarball.
+
The <tt>kernel</tt> directory consists of three main subdirectories: <tt>prebuilt</tt>, <tt>config</tt>, and <tt>tarball</tt>.
  
 
<pre>
 
<pre>
Line 28: Line 35:
 
</pre>
 
</pre>
  
−
The prebuilt directory contains prebuilt kernel images and kernel modules. While prebuilt ION kernel  
+
The <tt>prebuilt</tt> directory contains prebuilt kernel images and modules. While a complete prebuilt ION kernel ELF file is provided, for the CN kernel we provide intermediate object files instead. This is because we embed the CN ramdisk in the CN kernel image when building ZeptoOS, and this process requires the object files.
−
ELF file is found, no CN kernel ELF file is found here. This is because CN kernel ramdisk is embedded in kernel  
+
 
−
and we wanted to provide a way to replace the CNK ramdisk without invoking CN build process which requires  
+
The <tt>tarball</tt> directory contains kernel tarballs separately for the ION and the CN Linux kernel. Technically, those tarballs are snapshots of the ZeptoOS kernel git repository. The directory might contain a <tt>.patch</tt> file that contains the differences between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git for small modifications. Associated git log files can also be found in this directory. A <tt>.SNAPSHOT_HEAD</tt> file indicates the git revision at the time when a snapshot was created, so this information is used to create a patch file.
−
untar'ing source code. Instead CN kernel objects are found in prebuilt directory and
+
 
−
a CN kernel ELF is created from the objects and ramdisk every time when you invoke certain make target.
+
As an example, here is a list of files for the CN kernel:
  
−
The tarball directory contains kernel tarballs separately for ION and CN Linux kernel. Technically, those tarball are a snopshot of
 
−
the Zepto kernel git repository. The directory might contain a .patch file that contains difference between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git even for small modification.
 
−
Associated git log file can be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot is created, so this information is used to create a patch file.
 
 
<pre>
 
<pre>
 
linux-2.6.19.2-BGP-V1R3.git.log
 
linux-2.6.19.2-BGP-V1R3.git.log
Line 44: Line 48:
 
</pre>
 
</pre>
  
−
The config directory contains Linux kernel config.  A config file is basically associated to a tarball by kernel version.
+
The <tt>config</tt> directory contains Linux kernel configs.  In case of the 2.6.19 kernel, we provide separate config files for the compute node and the I/O node.
−
i.e, bgp-cn-2.6.19.2-dot-config for linux-2.6.19.2-BGP-V1R3.tar.bz2
+
 
−
See also Makefile.
+
==Building a kernel==
  
−
==Build kernel==
+
The <tt>Makefile</tt> in the <tt>kernel</tt> directory has many targets. Just type <tt>make</tt> and it will print out a help message.
  
−
Makefile in the kernel directory has many options. Just type 'make' it will show you help message.  
+
If one needs to build (or rebuild) a kernel from a source tarball, use <tt>bgp-ion-linux-build</tt> or <tt>bgp-cn-linux-build</tt> targets. By default, it extracts ION or CN kernel tarball in a directory named <tt>work</tt>, applies a patch if any and starts the kernel build. Once the kernel has successfully been built, kernel images (in both ZeptoOS top-level directory and the <tt>tmp</tt> directory) will be replaced with newly built ones. The ION kernel source code is extracted into <tt>work/linux-2.6.16.46-297-BGP-V1R3</tt> and the CN kernel source into <tt>work/linux-2.6.19.2-BGP-V1R3</tt>.
  
−
If you need to build (or rebuild) kernel from the tarball of kernel source code, use '''bgp-ion-linux-build''' or '''bgp-cn-linux-build target'''.
+
Here is an example of building and rebuilding the CN kernel:
−
By default it extracts ION or CN kernel tarball in a directory named work, apply a patch if any
 
−
and start kernel build. Once kernel has successfully been built, kernel images
 
−
(in both Zepto top directory and tmp directory) will be replaced with newly built images.
 
−
ION kernel source code is extracted in '''work/linux-2.6.16.46-297-BGP-V1R3''' and
 
−
CN kernel source is in '''work/linux-2.6.19.2-BGP-V1R3'''.
 
−
Here is an example of build, rebuilding kernel.
 
  
 
<pre>
 
<pre>
Line 71: Line 69:
 
</pre>
 
</pre>
  
−
===Build kernel from Zepto kernel git repo===
+
===Building a kernel from the ZeptoOS kernel git repository===
−
As mentioned above the kernel tarball is used as source code by default.  
+
 
−
If you specify '''GIT=1''' to the make command,  
+
As mentioned earlier, kernel tarballs are used as the source by default. If instead one passes <tt>GIT=1</tt> to <tt>make</tt>, one can build directly from the ZeptoOS kernel git tree. This is very useful for kernel development since it makes it easier to keep track of local modifications.  
−
you can directly build from our Zepto kernel git tree. This is very useful for kernel development since  
 
−
you can keep track your modifications.  
 
  
 
<pre>
 
<pre>
Line 86: Line 82:
 
</pre>
 
</pre>
  
−
Note that repo/linux-2.6.19.2-BGP-V1R3 is a cloned git repository that is cloned from
+
This will create <tt>repo/linux-2.6.19.2-BGP-V1R3</tt>, which is a git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you cannot push your modifications to it. Instead, please post any patches to the [mailto:zeptoos@lists.mcs.anl.gov ZeptoOS developers mailing list].  
−
http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/.  
 
−
Our http repo is read-only, so you can't push your modifications to the http repo.  
 
−
You can post your git patch to [mailto:[email protected] ZeptoOS mailing list] first.  
 
  
−
See also [http://bg-linux.anl-external.org/wiki/index.php/Main_Page BG-Linux page] for our kernel git repo details.
+
See also the [http://bg-linux.anl-external.org/wiki/index.php/Main_Page BG-Linux page] for the details on our kernel git repository.
  
 
===Kernel config===
 
===Kernel config===
  
−
Initially, config/bgp-cn-2.6.19.2-dot-config is applied CN Linux kernel build tree
+
When one invokes <tt>make</tt> with a kernel build target for the first time, the associated kernel config file is copied to <tt>.config</tt> in the kernel build directory.  <tt>config/bgp-cn-2.6.19.2-dot-config</tt> is applied to the CN Linux kernel build tree, and <tt>config/bgp-ion-2.6.16.46-dot-config</tt> is applied to the ION Linux kernel build tree.
−
and config/bgp-ion-2.6.16.46-dot-config is applied to ION Linux kernel build tree.
 
−
Technically when you type a kernel build target the first time, associated kernel config file is copied to .config in
 
−
kernel build directory.  
 
  
−
Here is the location of kernel config file.
+
Here are the locations of the kernel config files:
 
* Regular build
 
* Regular build
 
** work/build-2.6.19.2-BGP-V1R3/.config
 
** work/build-2.6.19.2-BGP-V1R3/.config
Line 108: Line 98:
 
** repo/build-2.6.16.46-297-BGP-V1R3/.config
 
** repo/build-2.6.16.46-297-BGP-V1R3/.config
  
−
Please note that the kernel config file is copied only once until you do distclean or  
+
Please note that the kernel config file is copied only once, until one does a <tt>distclean</tt> or removes the files manually.
−
remove the files manually.
 
  
−
bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig target invoke text based Linux kernel configuration menu.
+
The <tt>bgp-cn-linux-menuconfig</tt> and <tt>bgp-ion-linux-menuconfig</tt> <tt>make</tt> targets invoke text-based Linux kernel configuration menus:
  
 
<pre>
 
<pre>
Line 118: Line 107:
 
</pre>
 
</pre>
  
−
For GIT build,
+
For GIT build:
 
<pre>
 
<pre>
 
$ make GIT=1 bgp-ion-linux-menuconfig
 
$ make GIT=1 bgp-ion-linux-menuconfig
Line 124: Line 113:
 
</pre>
 
</pre>
  
−
Those menu target never update default kernel config file in the config directory.  
+
These menu targets never update the default kernel config files from the <tt>config</tt> directory. If one wants to apply a new config permanently, please copy it to the <tt>config</tt> directory by hand:
−
If you want to apply new config permanently, please copy it to the config directory by hand.
 
−
For example,
 
 
<pre>
 
<pre>
−
$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config
+
$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config
 
</pre>
 
</pre>
  
−
==Kernel (command line) Parameters==
+
==Kernel (command line) parameters==
  
−
In usual server/desktop Linux environment, kernel parameters
+
In common server/desktop Linux environments, kernel parameters can be passed via a bootloader such as grub. However, Blue Gene/P boot mechanism does not provide such capability, so we have modified the CN Linux kernel (2.6.19) to use a kernel parameter string embedded in kernel ELF image file itself.
−
are passed via bootloader such as grub. However, BlueGene/P boot mechanism  
 
−
does not have such capability. Kernel parameters are very handy since we can skip kernel rebuild.
 
−
So we have modified CN Linux kernel (2.6.19) to allow to load kernel parameter string  
 
−
embedded in kernel ELF image file.  
 
  
−
You can add kernel parameters (or reset) to a kernel ELF file by
+
One can (re)set the kernel parameters in a kernel ELF file using a command line tool <tt>zkparam.py</tt>, located in the <tt>bin</tt> subdirectory of the ZeptoOS installation directory. Here is the synopsis of the tool:
−
a command line tool named zkparam.py, which is located in kernel directory.
 
−
Here is the synopsis of the tool.
 
  
 
<pre>
 
<pre>
−
zkparam.py KERNEL_ELF  [options]
+
zkparam.py <kernel_image> [options]
 
</pre>
 
</pre>
  
−
If you omit options, the tool shows you the current kernel parameters.
+
If options are omitted, the tool shows the current kernel parameters:
  
 
<pre>
 
<pre>
−
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
+
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
−
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf  
+
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf
 
Current Kernel Parameters:
 
Current Kernel Parameters:
 
  zepto_console_output=2
 
  zepto_console_output=2
 
</pre>
 
</pre>
  
−
===Zepto kernel specific kernel parameters===
+
===ZeptoOS-pecific kernel parameters===
  
−
* '''zepto_debug'''=INTEGER   
+
* '''zepto_debug'''=<integer>
−
** Specify the zepto kernel debug level
+
** Specifies the ZeptoOS kernel debug level.
−
** 0 turns off all zepto debug msgs. higher number more detail
+
** The higher the number, the more messages are generated.
 +
** <tt>0</tt> turns off all debug messages.
 
** default=1
 
** default=1
−
* '''flatmemsizeMB'''=INTEGER
+
* '''flatmemsizeMB'''=<integer>
−
** Specify the size of flatmemory in MB.
+
** Specifies the size of big memory in MB.
−
** Currently the granularity of memory size is limited to 256 MB
+
** Currently the granularity of memory size is 256&nbsp;MB.
 
** default=256  min=256  max=1792
 
** default=256  min=256  max=1792
−
* '''zepto_console_output'''=INTEGER
+
* '''zepto_console_output'''=<integer>
−
** Specify the console output behavior.  
+
** Specifies the console output behavior.  
−
** 0 disables all console output from kernel.  
+
** 0 disables console output from all compute nodes.
−
** 1 enables console output from one of nodes(personality rank=1)  
+
** 1 enables console output from the first compute node ([[FAQ#Torus rank|torus rank]] 0).
−
** 2 enables console output from all nodes
+
** 2 enables console output from all compute nodes.
−
 
 
−
==Log files,etc==
 
  
 +
==Log files, etc==
 +
<!--
 
===Compute Node log===
 
===Compute Node log===
  
Line 197: Line 179:
  
 
===RAS events===
 
===RAS events===
 +
-->
 +
 +
The compute node and I/O node logfiles have been discussed extensively in [[Testing#Log files|Testing]].
 +
 +
In addition to regular console logs, the kernels can also generate RAS messages, which will not appear in the log files; they are instead stored in a database on the service node. At Argonne we have a custom command line tool named <tt>bg-listevents</tt> that shows a record of RAS events (type <tt>bg-listevents -h</tt> for command line arguments).
  
−
RAS messages won't appear in the log files.
+
----
−
A command line tool named bg-listevents shows you a record of RAS events.
+
[[MPICH, DCMF, and SPI]] | [[ZeptoOS_Documentation|Top]] | [[Ramdisk]]
−
Type 'bg-listevents -h' for command line parameters.
 

Latest revision as of 13:58, 8 May 2009

MPICH, DCMF, and SPI | Top | Ramdisk

Introduction

We currently provide two Linux kernels:

  • 2.6.19-based kernel: ZeptoOS CN kernel
    • IBM V1R3 patch and ZeptoOS patch applied
    • 64 KB page size and big memory region available
    • Device drivers for compute node devices such as DMA, lockbox, etc
    • Allows to run MPICH/DCMF code through Zepto Compute Binary (ZCB)
    • Can be used as enhanced ION kernel
  • 2.6.16-based kernel: ZeptoOS ION kernel
    • IBM V1R3 patch applied
    • Only minor changes compared to the IBM ION kernel.

We focus our development efforts on the 2.6.19-based kernel. It is meant primarily for the compute nodes, but can also be used on the I/O nodes. The problem is that GPFS does not work with this kernel, so we also provide the 2.6.16-based kernel which GPFS does work with.

Kernel directory structure

The kernel directory consists of three main subdirectories: prebuilt, config, and tarball. 

kernel
|-- prebuilt
|   |-- 2.6.16
|   |   `-- ION
|   `-- 2.6.19
|       |-- CN
|       `-- objs
|-- tarball
`-- config

The prebuilt directory contains prebuilt kernel images and modules. While a complete prebuilt ION kernel ELF file is provided, for the CN kernel we provide intermediate object files instead. This is because we embed the CN ramdisk in the CN kernel image when building ZeptoOS, and this process requires the object files.

The tarball directory contains kernel tarballs separately for the ION and the CN Linux kernel. Technically, those tarballs are snapshots of the ZeptoOS kernel git repository. The directory might contain a .patch file that contains the differences between the last snapshot and the current git HEAD since we wanted to avoid creating a snapshot from git for small modifications. Associated git log files can also be found in this directory. A .SNAPSHOT_HEAD file indicates the git revision at the time when a snapshot was created, so this information is used to create a patch file.

As an example, here is a list of files for the CN kernel: 

linux-2.6.19.2-BGP-V1R3.git.log
linux-2.6.19.2-BGP-V1R3.patch
linux-2.6.19.2-BGP-V1R3.SNAPSHOT_HEAD
linux-2.6.19.2-BGP-V1R3.tar.bz2

The config directory contains Linux kernel configs. In case of the 2.6.19 kernel, we provide separate config files for the compute node and the I/O node.

Building a kernel

The Makefile in the kernel directory has many targets. Just type make and it will print out a help message.

If one needs to build (or rebuild) a kernel from a source tarball, use bgp-ion-linux-build or bgp-cn-linux-build targets. By default, it extracts ION or CN kernel tarball in a directory named work, applies a patch if any and starts the kernel build. Once the kernel has successfully been built, kernel images (in both ZeptoOS top-level directory and the tmp directory) will be replaced with newly built ones. The ION kernel source code is extracted into work/linux-2.6.16.46-297-BGP-V1R3 and the CN kernel source into work/linux-2.6.19.2-BGP-V1R3.

Here is an example of building and rebuilding the CN kernel: 

$ cd kernel
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf
$ vi work/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make bgp-cn-linux-build
....
$ ls -al ../BGP-CN-zImage-with-initrd.elf

Building a kernel from the ZeptoOS kernel git repository

As mentioned earlier, kernel tarballs are used as the source by default. If instead one passes GIT=1 to make, one can build directly from the ZeptoOS kernel git tree. This is very useful for kernel development since it makes it easier to keep track of local modifications. 

$ cd kernel
$ make GIT=1 bgp-cn-linux-build
....
$ vi repo/linux-2.6.19.2-BGP-V1R3/kernel/sched.c
$ make GIT=1 bgp-cn-linux-build
....

This will create repo/linux-2.6.19.2-BGP-V1R3, which is a git repository that is cloned from http://git.anl-external.org/bg-linux.repos/linux-2.6.19-BGP-V1R3.git/. Our http repo is read-only, so you cannot push your modifications to it. Instead, please post any patches to the ZeptoOS developers mailing list.

See also the BG-Linux page for the details on our kernel git repository.

Kernel config

When one invokes make with a kernel build target for the first time, the associated kernel config file is copied to .config in the kernel build directory. config/bgp-cn-2.6.19.2-dot-config is applied to the CN Linux kernel build tree, and config/bgp-ion-2.6.16.46-dot-config is applied to the ION Linux kernel build tree.

Here are the locations of the kernel config files:

  • Regular build
    • work/build-2.6.19.2-BGP-V1R3/.config
    • work/build-2.6.16.46-297-BGP-V1R3/.config
  • GIT build
    • repo/build-2.6.19.2-BGP-V1R3/.config
    • repo/build-2.6.16.46-297-BGP-V1R3/.config

Please note that the kernel config file is copied only once, until one does a distclean or removes the files manually.

The bgp-cn-linux-menuconfig and bgp-ion-linux-menuconfig make targets invoke text-based Linux kernel configuration menus: 

$ make bgp-ion-linux-menuconfig
$ make bgp-cn-linux-menuconfig

For GIT build: 

$ make GIT=1 bgp-ion-linux-menuconfig
$ make GIT=1 bgp-cn-linux-menuconfig

These menu targets never update the default kernel config files from the config directory. If one wants to apply a new config permanently, please copy it to the config directory by hand: 

$ cp work/build-2.6.19.2-BGP-V1R3/.config config/bgp-cn-2.6.19.2-dot-config

Kernel (command line) parameters

In common server/desktop Linux environments, kernel parameters can be passed via a bootloader such as grub. However, Blue Gene/P boot mechanism does not provide such capability, so we have modified the CN Linux kernel (2.6.19) to use a kernel parameter string embedded in kernel ELF image file itself.

One can (re)set the kernel parameters in a kernel ELF file using a command line tool zkparam.py, located in the bin subdirectory of the ZeptoOS installation directory. Here is the synopsis of the tool: 

zkparam.py <kernel_image> [options]

If options are omitted, the tool shows the current kernel parameters: 

$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf zepto_console_output=2
$ ./kernel/zkparam.py BGP-CN-zImage-with-initrd.elf
Current Kernel Parameters:
 zepto_console_output=2

ZeptoOS-pecific kernel parameters

  • zepto_debug=<integer>
    • Specifies the ZeptoOS kernel debug level.
    • The higher the number, the more messages are generated.
    • 0 turns off all debug messages.
    • default=1
  • flatmemsizeMB=<integer>
    • Specifies the size of big memory in MB.
    • Currently the granularity of memory size is 256 MB.
    • default=256 min=256 max=1792
  • zepto_console_output=<integer>
    • Specifies the console output behavior.
    • 0 disables console output from all compute nodes.
    • 1 enables console output from the first compute node (torus rank 0).
    • 2 enables console output from all compute nodes.

Log files, etc

The compute node and I/O node logfiles have been discussed extensively in Testing.

In addition to regular console logs, the kernels can also generate RAS messages, which will not appear in the log files; they are instead stored in a database on the service node. At Argonne we have a custom command line tool named bg-listevents that shows a record of RAS events (type bg-listevents -h for command line arguments).

MPICH, DCMF, and SPI | Top | Ramdisk

Retrieved from "http:/// /zeptoos/index.php?title=Kernel&oldid=601"