Difference between revisions of "Installation"

From ZeptoOS
Jump to navigationJump to search
Line 86: Line 86:
 
; loader
 
; loader
 
: A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
 
: A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
−
; CN images
+
; CN image list
−
: A series of images loaded in order onto each compute node.
+
: A series of images loaded onto each compute node.
−
; ION images
+
; ION image list
−
: A series of images loaded in order onto each I/O node.
+
: A series of images loaded onto each I/O node.
  
−
All the images are loaded onto the machine from the service node via the service (JTAG) network.
+
All the images are loaded onto the machine from the service node via the service (JTAG) network.  The loader is loaded first, followed by the CN- and ION-specific images loaded in order.
  
−
; CNS
+
The CN image list defaults to CNS followed by the CNK.  The ION image list defaults to CNS, followed by the Linux kernel, followed by the Linux ramdisk.  CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.
−
: Common Node Services, a proprietary "firmware" loaded onto all compute and I/O nodes, which negotiates between the hardware and the kernel.
 
−
; CNK
 
−
: Compute node kernel
 
−
; INK
 
−
: I/O node kernel
 
−
; ramdisk
 
−
: I/O Node ramdisk
 
  
−
All these files are loaded onto the machine from the service node via the service (JTAG) network.  <tt>uloader</tt> and <tt>CNS</tt> are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
+
To enable ZeptoOS, we need to boot the Zepto CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use.  The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
  
−
The system allows us to specify multiple CN images. Command Node Service(CNS)
+
In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.
−
image and CN kernel image(IBM CNK) are default images and loaded into CN's main memory in order.
 
−
/bgsys/drivers/ppcfloor/boot/cns is only choice for CNS, which is not an open source code , either.
 
−
We can also specify multiple ION images. CNS, IBM ION Linux kernel image and ION Linux ramdisk are default images. They are also loaded into ION's main memory in order.  
 
  
−
To enable Zepto feature, we need to boot Zepto CN kernel and ION kernel in a partition that we use.
+
===System using Cobalt===
−
We describe how to assign and boot Zepto images in this section.
 
−
 
 
−
===Cobalt installed system===
 
−
 
−
If your BGP system has the cobalt scheduler installed and its kernel
 
−
profile feature has been configured properly, it would be easy to
 
−
boot Zepto kernel for your task.
 
 
   
 
   
−
What you need is to make a directory in the kernel profile directory
+
If the BGP system in question has the [http://trac.mcs.anl.gov/projects/cobalt/ Cobalt] scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS there should be easy.  All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.
−
and create a couple of symbolic link that point to Zepto images.
 
  
−
In ANL BGP, /bgsys/argonne-utils/profiles/ is the kernel profile directory.
+
On Argonne BGP machines, <tt>/bgsys/argonne-utils/profiles/</tt> is the top-level kernel profile directory; individual profiles are stored in its subdirectories.  Assuming that the ZeptoOS images have already been created (see [[Configuration]]), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:
−
Here are the concrete steps to create a new kernel profile.  Suppose that  
 
−
you have already built your Zepto kernel images and write permission to the kernel profile directory.
 
 
   
 
   
−
<pre>$ cd KERNEL_PROFILE_DIR
+
<pre>
−
$ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME
+
$ cd <kernel_profile_dir>
−
$ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf CNK  
+
$ mkdir <your_profile> && cd <your_profile>
−
$ ln -s ZEPTO_DIR/BGP-ION-zImage.elf INK  
+
$ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK
−
$ ln -s ZEPTO_DIR/BGP-ION-ramdisk-for-CNL.elf ramdisk  
+
$ ln -s <zepto_dir>/BGP-ION-zImage.elf INK
−
$ ln -s ../factory-default/CNS  
+
$ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk
−
$ ln -s ../factory-default/uloader  
+
$ ln -s ../default/CNS
 +
$ ln -s ../default/uloader
 
</pre>  
 
</pre>  
−
+
 
−
NOTE: your Zepto images must be readable from others, otherwise your
+
'''Note:''' ensure that the ZeptoOS images are world-readable, otherwise jobs will fail to start.  If needed, copy the images to the kernel profile rather than link to them.
−
job will fail. Please double check!!!
+
 
−
+
For Argonne users, we provide a convenience script <tt>mkprofile-ANL.sh</tt> with some extra features. The following command line is equivalent to the above steps:
−
For ANL user, we provide a convenient script named mkprofile-ANL.sh  
+
 
−
which essentially does what mentioned in above but has some extra  
+
<pre>
−
features. The following command line is equivalent to the steps  
+
$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>
−
described in above.
 
−
 
−
<pre>$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME
 
 
</pre>  
 
</pre>  
 
   
 
   
−
Invoking it with the -h option shows its help message. Use -c if you  
+
Invoking it with the <tt>-h</tt> option prints an overview of command line options. In particular, use <tt>-c</tt> if you prefer to copy images instead of making the links.
−
actually need to copy images instead of making symbolic link.  Use
 
−
-cn, -ion or -rd if you have a custom named image.  
 
 
   
 
   
−
<pre>$ ./mkprofile-ANL.sh -h  
+
<pre>
−
Usage: ./mkprofile-ANL.sh [OPTIONS]  
+
$ ./mkprofile-ANL.sh -h
−
+
Usage: ./mkprofile-ANL.sh [OPTIONS]
−
Options:  
+
 
−
-h            : Show this message  
+
Options:
−
-c            : Copy images instead of making symbolic link  
+
-h            : Show this message
−
-f            : Overwrite existing profile
+
-c            : Copy images instead of making symbolic link
−
--profile=name : Specify profile name  
+
-f            : Overwrite existing profile
−
--cn=fn        : Compute Node Kernel Image  
+
--profile=name : Specify profile name
−
--ion=fn      : Specify I/O Node Kernel Image      
+
--cn=fn        : Compute Node Kernel Image
−
--rd=fn        : Specify I/O Node Ramdisk Image  
+
--ion=fn      : Specify I/O Node Kernel Image
−
--ls          : show files in profile  
+
--rd=fn        : Specify I/O Node Ramdisk Image
−
--dryrun
+
--ls          : show files in profile
−
</pre>  
+
--dryrun
−
+
</pre>
−
Once you have properly setup your Zepto kernel profile, you can  
+
 
−
boot Zepto kernel by specifying your kernel profile name via the -k
+
Once the ZeptoOS kernel profile is set up, it can be booted
−
cobalt option.
+
by specifying the profile name when submitting a job using either <tt>cqsub</tt> or <tt>qsub</tt>:
 
   
 
   
−
<pre>$ cqsub -k YOUR_PROFILE_NAME ....  
+
<pre>
 +
$ cqsub -k <profile_name> ...
 +
$ qsub --kernel <profile_name> ...
 
</pre>
 
</pre>
  
−
===MMCS console===
+
===Manual installation using the MMCS console===
  
 
Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed.  
 
Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed.  

Revision as of 11:46, 6 May 2009

Configuration | Top | Testing

Installing the support files

Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) use primarily when building compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and type: 

$ python install.py /path/to/install

This will install the following files: 

Install Home:  /path/to/install
Creating install directories ...
Making /path/to/install/bin
Making /path/to/install/include
Making /path/to/install/lib

Installing mpich2 scripts ...
Installing /path/to/install/bin/zmpicc
Installing /path/to/install/bin/zmpicxx
Installing /path/to/install/bin/zmpif77
Installing /path/to/install/bin/zmpif90
Installing /path/to/install/bin/zmpixlc_r
Installing /path/to/install/bin/zmpixlc
Installing /path/to/install/bin/zmpixlcxx_r
Installing /path/to/install/bin/zmpixlcxx
Installing /path/to/install/bin/zmpixlf2003_r
Installing /path/to/install/bin/zmpixlf2003
Installing /path/to/install/bin/zmpixlf77_r
Installing /path/to/install/bin/zmpixlf77
Installing /path/to/install/bin/zmpixlf90_r
Installing /path/to/install/bin/zmpixlf90
Installing /path/to/install/bin/zmpixlf95_r
Installing /path/to/install/bin/zmpixlf95

Installing misc files ...
/path/to/install/bin/zelftool

Installing MPICH2 headers ...
/path/to/install/include/mpe_thread.h
/path/to/install/include/mpi_base.mod
/path/to/install/include/mpi_constants.mod
/path/to/install/include/mpicxx.h
/path/to/install/include/mpif.h
/path/to/install/include/mpi.h
/path/to/install/include/mpi.mod
/path/to/install/include/mpiof.h
/path/to/install/include/mpio.h
/path/to/install/include/mpi_sizeofs.mod
/path/to/install/include/mpix.h

Installing MPICH2 libraries ...
/path/to/install/lib/libcxxmpich.zcl.a
/path/to/install/lib/libfmpich_.zcl.a
/path/to/install/lib/libfmpich.zcl.a
/path/to/install/lib/libmpich.zcl.a
/path/to/install/lib/libmpich.zclf90.a

Installing SPI libraries ...
/path/to/install/lib/libSPI.zcl.a
/path/to/install/lib/libzcl.a

Installing Zoid files ...
/path/to/install/lib/libzoid_cn.a

Installing DCMF files ...
/path/to/install/include/dcmf_collectives.h
/path/to/install/include/dcmf_coremath.h
/path/to/install/include/dcmf_globalcollectives.h
/path/to/install/include/dcmf.h
/path/to/install/include/dcmf_multisend.h
/path/to/install/include/dcmf_optimath.h
/path/to/install/lib/libdcmfcoll.zcl.a
/path/to/install/lib/libdcmf.zcl.a

Setting up a kernel profile

The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.

Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units. More importantly, the system supports using different software stacks (what we call kernel profiles) on different partitions at the same time. This means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.

A kernel profile consists of the following elements:

loader
A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
CN image list
A series of images loaded onto each compute node.
ION image list
A series of images loaded onto each I/O node.

All the images are loaded onto the machine from the service node via the service (JTAG) network. The loader is loaded first, followed by the CN- and ION-specific images loaded in order.

The CN image list defaults to CNS followed by the CNK. The ION image list defaults to CNS, followed by the Linux kernel, followed by the Linux ramdisk. CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.

To enable ZeptoOS, we need to boot the Zepto CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.

In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.

System using Cobalt

If the BGP system in question has the Cobalt scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS there should be easy. All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.

On Argonne BGP machines, /bgsys/argonne-utils/profiles/ is the top-level kernel profile directory; individual profiles are stored in its subdirectories. Assuming that the ZeptoOS images have already been created (see Configuration), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile: 

$ cd <kernel_profile_dir>
$ mkdir <your_profile> && cd <your_profile>
$ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK
$ ln -s <zepto_dir>/BGP-ION-zImage.elf INK
$ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk
$ ln -s ../default/CNS
$ ln -s ../default/uloader

Note: ensure that the ZeptoOS images are world-readable, otherwise jobs will fail to start. If needed, copy the images to the kernel profile rather than link to them.

For Argonne users, we provide a convenience script mkprofile-ANL.sh with some extra features. The following command line is equivalent to the above steps: 

$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>

Invoking it with the -h option prints an overview of command line options. In particular, use -c if you prefer to copy images instead of making the links. 

$ ./mkprofile-ANL.sh -h
Usage: ./mkprofile-ANL.sh [OPTIONS]

Options:
-h             : Show this message
-c             : Copy images instead of making symbolic link
-f             : Overwrite existing profile
--profile=name : Specify profile name
--cn=fn        : Compute Node Kernel Image
--ion=fn       : Specify I/O Node Kernel Image
--rd=fn        : Specify I/O Node Ramdisk Image
--ls           : show files in profile
--dryrun

Once the ZeptoOS kernel profile is set up, it can be booted by specifying the profile name when submitting a job using either cqsub or qsub: 

$ cqsub -k <profile_name> ...
$ qsub --kernel <profile_name> ...

Manual installation using the MMCS console

Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed. If you don't have cobalt, probably we'll need to use Midplane Management Control System(MMCS). We explain how to assign and boot your own kernel images using MMCS.

Here is a brief summary of MMCS.

  • Available on all BGP systems
  • the lowest control mechanism for BGP partition
  • allocate, free or query of block(partition)
  • status check
  • assign boot images
  • low level debug command

Due to its low level interface, it requires administrator level permission to use it. You also need to reserve a partition (or block).


Assign Zepto images to a BGP partition

Login to the service node and start MMCS. 

$ ssh sn         
sn $ ./mmcs.sh  

[mmcs.sh] 
#!/bin/sh 
 
export DB2HOME=/dbhome/bgpdb2c/sqllib 
DB2SRC=${DB2HOME}/db2profile 
[ -f "$DB2SRC" ] && . $DB2SRC 
 
cd /bgsys/drivers/ppcfloor/bin 
./mmcs_db_console

Please memorize that the current configuration. You need to revert the blockinfo to the original configuration later on after you are done with Zepto kernel. 

console $ set_username YOUR_LOGIN_NAME 
console $ getblockinfo BGP_BLOCK_NAME 
OK 
boot info for block BGP_BLOCK_NAME: 
mloader: /bgsys/drivers/ppcfloor/boot/uloader 
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk 
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk 
status: F

Assign Zepto images to a parition 

console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,BGP_CN_LINUX_KERNEL_PATH /bgsys/drivers/ppcfloor\
/boot/cns,BGP_ION_LINUX_KERNEL_PATH,BGP_ION_LINUX_RAMDISK_PATH 
console $ quit

Boot Zepto kernel

Once you have configured a partition with Zepto kernels correctly, Zepto kernels will be booted when you run a job on that partition(via mpirun for example) 

fen $ mpirun -verbose 1 -partition BGP_BLOCK_NAME  -np 64 -timeout 600 -cwd `pwd` -exe ./a.out

Restore to the original configuration(Don't forget!!!)

After you have done your work on Zepto kernel, you need to restore to the original configuration. Here is an example. 

fen $ ssh sn 
sn $ ./mmcs.sh 
console $ set_username YOUR_LOGIN_NAME 
console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk /bgsys/drivers/\
ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk 
console $ quit

Configuration | Top | Testing

Retrieved from "http:/// /zeptoos/index.php?title=Installation&oldid=550"