Difference between revisions of "Kernel Profile"

From ZeptoOS
Jump to navigationJump to search
 
(14 intermediate revisions by one other user not shown)
Line 1: Line 1:
−
BGP is capble of running different kernel per partition.  By default
+
[[Configuration]] | [[ZeptoOS_Documentation|Top]] | [[Testing]]
−
IBM CNK and ION kernel(Linux) is booted in a partition when you submit
+
----
−
a job. To enable Zepto feature, you need to configure the system
+
 
−
properly with Zepto Kernel
+
==Installing the support files==
−
+
 
−
===Cobalt installed system===
+
In order to install the headers, libraries and the binaries, just type:
 +
<pre>
 +
$ python install.py /path/to/install
 +
</pre>
 +
 
 +
This will install the following files:
 +
<pre>
 +
Install Home:  /path/to/install
 +
Creating install directories ...
 +
Making /path/to/install/bin
 +
Making /path/to/install/include
 +
Making /path/to/install/lib
 +
 
 +
Installing mpich2 scripts ...
 +
Installing /path/to/install/bin/zmpicc
 +
Installing /path/to/install/bin/zmpicxx
 +
Installing /path/to/install/bin/zmpif77
 +
Installing /path/to/install/bin/zmpif90
 +
Installing /path/to/install/bin/zmpixlc_r
 +
Installing /path/to/install/bin/zmpixlc
 +
Installing /path/to/install/bin/zmpixlcxx_r
 +
Installing /path/to/install/bin/zmpixlcxx
 +
Installing /path/to/install/bin/zmpixlf2003_r
 +
Installing /path/to/install/bin/zmpixlf2003
 +
Installing /path/to/install/bin/zmpixlf77_r
 +
Installing /path/to/install/bin/zmpixlf77
 +
Installing /path/to/install/bin/zmpixlf90_r
 +
Installing /path/to/install/bin/zmpixlf90
 +
Installing /path/to/install/bin/zmpixlf95_r
 +
Installing /path/to/install/bin/zmpixlf95
 +
 
 +
Installing misc files ...
 +
/path/to/install/bin/zelftool
 +
 
 +
Installing MPICH2 headers ...
 +
/path/to/install/include/mpe_thread.h
 +
/path/to/install/include/mpi_base.mod
 +
/path/to/install/include/mpi_constants.mod
 +
/path/to/install/include/mpicxx.h
 +
/path/to/install/include/mpif.h
 +
/path/to/install/include/mpi.h
 +
/path/to/install/include/mpi.mod
 +
/path/to/install/include/mpiof.h
 +
/path/to/install/include/mpio.h
 +
/path/to/install/include/mpi_sizeofs.mod
 +
/path/to/install/include/mpix.h
 +
 
 +
Installing MPICH2 libraries ...
 +
/path/to/install/lib/libcxxmpich.zcl.a
 +
/path/to/install/lib/libfmpich_.zcl.a
 +
/path/to/install/lib/libfmpich.zcl.a
 +
/path/to/install/lib/libmpich.zcl.a
 +
/path/to/install/lib/libmpich.zclf90.a
 +
 
 +
Installing SPI libraries ...
 +
/path/to/install/lib/libSPI.zcl.a
 +
/path/to/install/lib/libzcl.a
 +
 
 +
Installing Zoid files ...
 +
/path/to/install/lib/libzoid_cn.a
 +
 
 +
Installing DCMF files ...
 +
/path/to/install/include/dcmf_collectives.h
 +
/path/to/install/include/dcmf_coremath.h
 +
/path/to/install/include/dcmf_globalcollectives.h
 +
/path/to/install/include/dcmf.h
 +
/path/to/install/include/dcmf_multisend.h
 +
/path/to/install/include/dcmf_optimath.h
 +
/path/to/install/lib/libdcmfcoll.zcl.a
 +
/path/to/install/lib/libdcmf.zcl.a
 +
</pre>
 +
 
 +
==Setting up a kernel profile==
 +
 
 +
The BlueGene/P system is capble of assigning different boot images per partition.   
 +
The system allows us to specify loader program, CN images and ION images.
 +
The loader program is loaded into main memory via jtag network and executed first.
 +
/bgsys/drivers/ppcfloor/boot/uloader is only loader program, which is not an open source code.
 +
The system allows us to specify multiple CN images. Command Node Service(CNS)
 +
image and CN kernel image(IBM CNK) are default images and loaded into CN's main memory in order.
 +
/bgsys/drivers/ppcfloor/boot/cns is only choice for CNS, which is not an open source code , either.
 +
We can also specify multiple ION images. CNS, IBM ION Linux kernel image and ION Linux ramdisk are default images. They are also loaded into ION's main memory in order.
 +
 
 +
To enable Zepto feature, we need to boot Zepto CN kernel and ION kernel in a partition that we use.
 +
We describe how to assign and boot Zepto images in this section.
 +
 
 +
===Cobalt installed system===
 
   
 
   
 
If your BGP system has the cobalt scheduler installed and its kernel  
 
If your BGP system has the cobalt scheduler installed and its kernel  
 
profile feature has been configured properly, it would be easy to  
 
profile feature has been configured properly, it would be easy to  
−
boot Zepto kernel.  
+
boot Zepto kernel for your task.
 
   
 
   
−
You can find a directory called kernel profile directory on login
+
What you need is to make a directory in the kernel profile directory
−
nodes (/bgsys/argonne-utils/profiles/ in ANL BGP system for
+
and create a couple of symbolic link that point to Zepto images.
−
example). Here are step to create a new kernel profile.  Suppose that  
+
 
−
you have already built your Zepto kernel images.  
+
In ANL BGP, /bgsys/argonne-utils/profiles/ is the kernel profile directory.
 +
Here are the concrete steps to create a new kernel profile.  Suppose that  
 +
you have already built your Zepto kernel images and write permission to the kernel profile directory.
 
   
 
   
−
<pre> $ cd KERNEL_PROFILE_DIR  
+
<pre>$ cd KERNEL_PROFILE_DIR  
 
$ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME  
 
$ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME  
 
$ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf  CNK  
 
$ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf  CNK  
Line 24: Line 112:
 
</pre>  
 
</pre>  
 
   
 
   
−
NOTE: Your Zepto images must be readable from others, otherwise your  
+
NOTE: your Zepto images must be readable from others, otherwise your  
−
job will fail. Please double check!  
+
job will fail. Please double check!!!
 
   
 
   
 
For ANL user, we provide a convenient script named mkprofile-ANL.sh  
 
For ANL user, we provide a convenient script named mkprofile-ANL.sh  
−
which essentially does what mentioned in above but has some other
+
which essentially does what mentioned in above but has some extra
−
features. The following commen line is equivalent to the steps  
+
features. The following command line is equivalent to the steps  
−
decribed in above.  
+
described in above.  
 
   
 
   
 
<pre>$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME  
 
<pre>$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME  
 
</pre>  
 
</pre>  
 
   
 
   
−
Invoking it with the -h option shows help message. Use -c if you  
+
Invoking it with the -h option shows its help message. Use -c if you  
 
actually need to copy images instead of making symbolic link.  Use  
 
actually need to copy images instead of making symbolic link.  Use  
 
-cn, -ion or -rd if you have a custom named image.  
 
-cn, -ion or -rd if you have a custom named image.  
Line 45: Line 133:
 
-h            : Show this message  
 
-h            : Show this message  
 
-c            : Copy images instead of making symbolic link  
 
-c            : Copy images instead of making symbolic link  
−
-f            : Overwrite exsiting profile   
+
-f            : Overwrite existing profile   
 
--profile=name : Specify profile name  
 
--profile=name : Specify profile name  
 
--cn=fn        : Compute Node Kernel Image   
 
--cn=fn        : Compute Node Kernel Image   
Line 54: Line 142:
 
</pre>  
 
</pre>  
 
   
 
   
−
Once you have properly configured your Zepto kernel profile, you can  
+
Once you have properly setup your Zepto kernel profile, you can  
 
boot Zepto kernel by specifying your kernel profile name via the -k  
 
boot Zepto kernel by specifying your kernel profile name via the -k  
 
cobalt option.  
 
cobalt option.  
 
   
 
   
 
<pre>$ cqsub -k YOUR_PROFILE_NAME ....  
 
<pre>$ cqsub -k YOUR_PROFILE_NAME ....  
−
</pre>  
+
</pre>
−
+
 
−
==MMCS console==
+
===MMCS console===
 +
 
 +
Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed.
 +
If you don't have cobalt, probably we'll need to use Midplane Management Control System(MMCS).
 +
We explain how to assign and boot your own kernel images using MMCS.
 +
 
 +
Here is a brief summary of MMCS.
 +
* Available on all BGP systems
 +
* the lowest control mechanism for BGP partition
 +
* allocate, free or query of block(partition)
 +
* status check
 +
* assign boot images
 +
* low level debug command
 +
 
 +
Due to its low level interface, it requires administrator level permission to use it.
 +
You also need to reserve a partition (or block).
 +
 
 +
 
 +
====Assign Zepto images to a BGP partition====
 
   
 
   
−
If no cobalt kernel profile feature is available on your BGP system,
+
Login to the service node and start MMCS.
−
using MMCS console is choice. What you basically do by mmcs console is
 
−
to assign Zepto kernels statatically to a partition you use.
 
−
 
−
 
−
===Assign Zepto images to a BGP partition===
 
−
 
−
Login to the service node and start MMCS  
 
 
   
 
   
 
<pre>$ ssh sn         
 
<pre>$ ssh sn         
Line 87: Line 186:
 
</pre>  
 
</pre>  
 
   
 
   
−
Memorize that the current configuration. You need to revert the  
+
Please memorize that the current configuration. You need to revert the  
−
blockinfo to the original configuration after you have done using
+
blockinfo to the original configuration later on after you are done with Zepto kernel.  
−
Zepto kernel.  
 
 
   
 
   
 
<pre>console $ set_username YOUR_LOGIN_NAME  
 
<pre>console $ set_username YOUR_LOGIN_NAME  
Line 108: Line 206:
 
</pre>  
 
</pre>  
  
−
===Boot Zepto kernel===  
+
====Boot Zepto kernel====
 
   
 
   
 
Once you have configured a partition with Zepto kernels correctly,  
 
Once you have configured a partition with Zepto kernels correctly,  
Line 117: Line 215:
 
</pre>  
 
</pre>  
 
   
 
   
−
===Restore to the original configuration(Don't forget!!!)===  
+
====Restore to the original configuration(Don't forget!!!)====
 
   
 
   
 
After you have done your work on Zepto kernel, you need to restore to  
 
After you have done your work on Zepto kernel, you need to restore to  
Line 129: Line 227:
 
console $ quit  
 
console $ quit  
 
</pre>
 
</pre>
 +
 +
----
 +
[[Configuration]] | [[ZeptoOS_Documentation|Top]] | [[Testing]]

Latest revision as of 13:53, 5 May 2009

Configuration | Top | Testing

Installing the support files

In order to install the headers, libraries and the binaries, just type: 

$ python install.py /path/to/install

This will install the following files: 

Install Home:  /path/to/install
Creating install directories ...
Making /path/to/install/bin
Making /path/to/install/include
Making /path/to/install/lib

Installing mpich2 scripts ...
Installing /path/to/install/bin/zmpicc
Installing /path/to/install/bin/zmpicxx
Installing /path/to/install/bin/zmpif77
Installing /path/to/install/bin/zmpif90
Installing /path/to/install/bin/zmpixlc_r
Installing /path/to/install/bin/zmpixlc
Installing /path/to/install/bin/zmpixlcxx_r
Installing /path/to/install/bin/zmpixlcxx
Installing /path/to/install/bin/zmpixlf2003_r
Installing /path/to/install/bin/zmpixlf2003
Installing /path/to/install/bin/zmpixlf77_r
Installing /path/to/install/bin/zmpixlf77
Installing /path/to/install/bin/zmpixlf90_r
Installing /path/to/install/bin/zmpixlf90
Installing /path/to/install/bin/zmpixlf95_r
Installing /path/to/install/bin/zmpixlf95

Installing misc files ...
/path/to/install/bin/zelftool

Installing MPICH2 headers ...
/path/to/install/include/mpe_thread.h
/path/to/install/include/mpi_base.mod
/path/to/install/include/mpi_constants.mod
/path/to/install/include/mpicxx.h
/path/to/install/include/mpif.h
/path/to/install/include/mpi.h
/path/to/install/include/mpi.mod
/path/to/install/include/mpiof.h
/path/to/install/include/mpio.h
/path/to/install/include/mpi_sizeofs.mod
/path/to/install/include/mpix.h

Installing MPICH2 libraries ...
/path/to/install/lib/libcxxmpich.zcl.a
/path/to/install/lib/libfmpich_.zcl.a
/path/to/install/lib/libfmpich.zcl.a
/path/to/install/lib/libmpich.zcl.a
/path/to/install/lib/libmpich.zclf90.a

Installing SPI libraries ...
/path/to/install/lib/libSPI.zcl.a
/path/to/install/lib/libzcl.a

Installing Zoid files ...
/path/to/install/lib/libzoid_cn.a

Installing DCMF files ...
/path/to/install/include/dcmf_collectives.h
/path/to/install/include/dcmf_coremath.h
/path/to/install/include/dcmf_globalcollectives.h
/path/to/install/include/dcmf.h
/path/to/install/include/dcmf_multisend.h
/path/to/install/include/dcmf_optimath.h
/path/to/install/lib/libdcmfcoll.zcl.a
/path/to/install/lib/libdcmf.zcl.a

Setting up a kernel profile

The BlueGene/P system is capble of assigning different boot images per partition. The system allows us to specify loader program, CN images and ION images. The loader program is loaded into main memory via jtag network and executed first. /bgsys/drivers/ppcfloor/boot/uloader is only loader program, which is not an open source code. The system allows us to specify multiple CN images. Command Node Service(CNS) image and CN kernel image(IBM CNK) are default images and loaded into CN's main memory in order. /bgsys/drivers/ppcfloor/boot/cns is only choice for CNS, which is not an open source code , either. We can also specify multiple ION images. CNS, IBM ION Linux kernel image and ION Linux ramdisk are default images. They are also loaded into ION's main memory in order.

To enable Zepto feature, we need to boot Zepto CN kernel and ION kernel in a partition that we use. We describe how to assign and boot Zepto images in this section.

Cobalt installed system

If your BGP system has the cobalt scheduler installed and its kernel profile feature has been configured properly, it would be easy to boot Zepto kernel for your task.

What you need is to make a directory in the kernel profile directory and create a couple of symbolic link that point to Zepto images.

In ANL BGP, /bgsys/argonne-utils/profiles/ is the kernel profile directory. Here are the concrete steps to create a new kernel profile. Suppose that you have already built your Zepto kernel images and write permission to the kernel profile directory. 

$ cd KERNEL_PROFILE_DIR 
$ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME 
$ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf  CNK 
$ ln -s ZEPTO_DIR/BGP-ION-zImage.elf  INK 
$ ln -s ZEPTO_DIR/BGP-ION-ramdisk-for-CNL.elf ramdisk 
$ ln -s ../factory-default/CNS 
$ ln -s ../factory-default/uloader

NOTE: your Zepto images must be readable from others, otherwise your job will fail. Please double check!!!

For ANL user, we provide a convenient script named mkprofile-ANL.sh which essentially does what mentioned in above but has some extra features. The following command line is equivalent to the steps described in above. 

$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME

Invoking it with the -h option shows its help message. Use -c if you actually need to copy images instead of making symbolic link. Use -cn, -ion or -rd if you have a custom named image. 

$ ./mkprofile-ANL.sh -h 
Usage: ./mkprofile-ANL.sh [OPTIONS]   
 
Options: 
-h             : Show this message 
-c             : Copy images instead of making symbolic link 
-f             : Overwrite existing profile  
--profile=name : Specify profile name 
--cn=fn        : Compute Node Kernel Image   
--ion=fn       : Specify I/O Node Kernel Image       
--rd=fn        : Specify I/O Node Ramdisk Image 
--ls           : show files in profile 
--dryrun

Once you have properly setup your Zepto kernel profile, you can boot Zepto kernel by specifying your kernel profile name via the -k cobalt option. 

$ cqsub -k YOUR_PROFILE_NAME ....

MMCS console

Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed. If you don't have cobalt, probably we'll need to use Midplane Management Control System(MMCS). We explain how to assign and boot your own kernel images using MMCS.

Here is a brief summary of MMCS.

  • Available on all BGP systems
  • the lowest control mechanism for BGP partition
  • allocate, free or query of block(partition)
  • status check
  • assign boot images
  • low level debug command

Due to its low level interface, it requires administrator level permission to use it. You also need to reserve a partition (or block).


Assign Zepto images to a BGP partition

Login to the service node and start MMCS. 

$ ssh sn         
sn $ ./mmcs.sh  

[mmcs.sh] 
#!/bin/sh 
 
export DB2HOME=/dbhome/bgpdb2c/sqllib 
DB2SRC=${DB2HOME}/db2profile 
[ -f "$DB2SRC" ] && . $DB2SRC 
 
cd /bgsys/drivers/ppcfloor/bin 
./mmcs_db_console

Please memorize that the current configuration. You need to revert the blockinfo to the original configuration later on after you are done with Zepto kernel. 

console $ set_username YOUR_LOGIN_NAME 
console $ getblockinfo BGP_BLOCK_NAME 
OK 
boot info for block BGP_BLOCK_NAME: 
mloader: /bgsys/drivers/ppcfloor/boot/uloader 
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk 
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk 
status: F

Assign Zepto images to a parition 

console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,BGP_CN_LINUX_KERNEL_PATH /bgsys/drivers/ppcfloor\
/boot/cns,BGP_ION_LINUX_KERNEL_PATH,BGP_ION_LINUX_RAMDISK_PATH 
console $ quit

Boot Zepto kernel

Once you have configured a partition with Zepto kernels correctly, Zepto kernels will be booted when you run a job on that partition(via mpirun for example) 

fen $ mpirun -verbose 1 -partition BGP_BLOCK_NAME  -np 64 -timeout 600 -cwd `pwd` -exe ./a.out

Restore to the original configuration(Don't forget!!!)

After you have done your work on Zepto kernel, you need to restore to the original configuration. Here is an example. 

fen $ ssh sn 
sn $ ./mmcs.sh 
console $ set_username YOUR_LOGIN_NAME 
console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk /bgsys/drivers/\
ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk 
console $ quit

Configuration | Top | Testing

Retrieved from "http:/// /zeptoos/index.php?title=Kernel_Profile&oldid=520"