Difference between revisions of "Installation"

From ZeptoOS
Jump to navigationJump to search
 
(6 intermediate revisions by the same user not shown)
Line 4: Line 4:
 
==Installing the support files==
 
==Installing the support files==
  
−
Installing ZeptoOS consists of two parts.  In the first part we install support files (header files, libraries, scripts, etc) use primarily when building compute node binaries for ZeptoOS.  To perform this step, change to the top-level directory and type:
+
Installing ZeptoOS consists of two parts.  In the first part we install support files (header files, libraries, scripts, etc) used primarily when building compute node binaries for ZeptoOS.  To perform this step, change to the top-level directory and type:
 
<pre>
 
<pre>
 
$ python install.py /path/to/install
 
$ python install.py /path/to/install
Line 14: Line 14:
 
Creating install directories ...
 
Creating install directories ...
 
Making /path/to/install/bin
 
Making /path/to/install/bin
 +
Making /path/to/install/cnbin
 
Making /path/to/install/include
 
Making /path/to/install/include
 
Making /path/to/install/lib
 
Making /path/to/install/lib
 +
Making /path/to/install/lib/zoid
  
−
Installing mpich2 scripts ...
+
Installing MPICH2 scripts ...
−
Installing /path/to/install/bin/zmpicc
+
/path/to/install/bin/zmpicc
−
Installing /path/to/install/bin/zmpicxx
+
/path/to/install/bin/zmpicxx
−
Installing /path/to/install/bin/zmpif77
+
/path/to/install/bin/zmpif77
−
Installing /path/to/install/bin/zmpif90
+
/path/to/install/bin/zmpif90
−
Installing /path/to/install/bin/zmpixlc_r
+
/path/to/install/bin/zmpixlc
−
Installing /path/to/install/bin/zmpixlc
+
/path/to/install/bin/zmpixlc_r
−
Installing /path/to/install/bin/zmpixlcxx_r
+
/path/to/install/bin/zmpixlcxx
−
Installing /path/to/install/bin/zmpixlcxx
+
/path/to/install/bin/zmpixlcxx_r
−
Installing /path/to/install/bin/zmpixlf2003_r
+
/path/to/install/bin/zmpixlf2003
−
Installing /path/to/install/bin/zmpixlf2003
+
/path/to/install/bin/zmpixlf2003_r
−
Installing /path/to/install/bin/zmpixlf77_r
+
/path/to/install/bin/zmpixlf77
−
Installing /path/to/install/bin/zmpixlf77
+
/path/to/install/bin/zmpixlf77_r
−
Installing /path/to/install/bin/zmpixlf90_r
+
/path/to/install/bin/zmpixlf90
−
Installing /path/to/install/bin/zmpixlf90
+
/path/to/install/bin/zmpixlf90_r
−
Installing /path/to/install/bin/zmpixlf95_r
+
/path/to/install/bin/zmpixlf95
−
Installing /path/to/install/bin/zmpixlf95
+
/path/to/install/bin/zmpixlf95_r
 +
 
 +
Installing Compute Node binaries ...
 +
/path/to/install/cnbin/cn-ipfwd.sh
 +
/path/to/install/cnbin/cn-ipfwd
  
 
Installing misc files ...
 
Installing misc files ...
 
/path/to/install/bin/zelftool
 
/path/to/install/bin/zelftool
 +
/path/to/install/bin/zkparam.py
  
 
Installing MPICH2 headers ...
 
Installing MPICH2 headers ...
 
/path/to/install/include/mpe_thread.h
 
/path/to/install/include/mpe_thread.h
 +
/path/to/install/include/mpi.h
 +
/path/to/install/include/mpi.mod
 
/path/to/install/include/mpi_base.mod
 
/path/to/install/include/mpi_base.mod
 
/path/to/install/include/mpi_constants.mod
 
/path/to/install/include/mpi_constants.mod
 +
/path/to/install/include/mpi_sizeofs.mod
 
/path/to/install/include/mpicxx.h
 
/path/to/install/include/mpicxx.h
 
/path/to/install/include/mpif.h
 
/path/to/install/include/mpif.h
−
/path/to/install/include/mpi.h
+
/path/to/install/include/mpio.h
−
/path/to/install/include/mpi.mod
 
 
/path/to/install/include/mpiof.h
 
/path/to/install/include/mpiof.h
−
/path/to/install/include/mpio.h
 
−
/path/to/install/include/mpi_sizeofs.mod
 
 
/path/to/install/include/mpix.h
 
/path/to/install/include/mpix.h
  
 
Installing MPICH2 libraries ...
 
Installing MPICH2 libraries ...
 
/path/to/install/lib/libcxxmpich.zcl.a
 
/path/to/install/lib/libcxxmpich.zcl.a
 +
/path/to/install/lib/libfmpich.zcl.a
 
/path/to/install/lib/libfmpich_.zcl.a
 
/path/to/install/lib/libfmpich_.zcl.a
−
/path/to/install/lib/libfmpich.zcl.a
 
 
/path/to/install/lib/libmpich.zcl.a
 
/path/to/install/lib/libmpich.zcl.a
 
/path/to/install/lib/libmpich.zclf90.a
 
/path/to/install/lib/libmpich.zclf90.a
Line 62: Line 69:
 
/path/to/install/lib/libzcl.a
 
/path/to/install/lib/libzcl.a
  
−
Installing Zoid files ...
+
Installing ZOID files ...
 
/path/to/install/lib/libzoid_cn.a
 
/path/to/install/lib/libzoid_cn.a
 +
/path/to/install/lib/zoid/libc.a
 +
/path/to/install/lib/zoid/libpthread.a
  
 
Installing DCMF files ...
 
Installing DCMF files ...
 +
/path/to/install/include/dcmf.h
 
/path/to/install/include/dcmf_collectives.h
 
/path/to/install/include/dcmf_collectives.h
 
/path/to/install/include/dcmf_coremath.h
 
/path/to/install/include/dcmf_coremath.h
 
/path/to/install/include/dcmf_globalcollectives.h
 
/path/to/install/include/dcmf_globalcollectives.h
−
/path/to/install/include/dcmf.h
 
 
/path/to/install/include/dcmf_multisend.h
 
/path/to/install/include/dcmf_multisend.h
 
/path/to/install/include/dcmf_optimath.h
 
/path/to/install/include/dcmf_optimath.h
 +
/path/to/install/lib/libdcmf.zcl.a
 
/path/to/install/lib/libdcmfcoll.zcl.a
 
/path/to/install/lib/libdcmfcoll.zcl.a
−
/path/to/install/lib/libdcmf.zcl.a
 
 
</pre>
 
</pre>
  
Line 80: Line 89:
 
The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.
 
The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.
  
−
Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units.  More importantly, the system supports using different software stacks (what we call ''kernel profiles'') on different partitions at the same time.  This means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.
+
Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units.  More importantly, the system supports using different software stacks (what we call ''kernel profiles'') on different partitions at the same time.  For our immediate purposes, this means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.
  
 
A kernel profile consists of the following elements:
 
A kernel profile consists of the following elements:
Line 86: Line 95:
 
; loader
 
; loader
 
: A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
 
: A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
−
; CN images
+
; CN image list
−
: A series of images loaded in order onto each compute node.
+
: A series of images loaded onto each compute node.
−
; ION images
+
; ION image list
−
: A series of images loaded in order onto each I/O node.
+
: A series of images loaded onto each I/O node.
−
 
 
−
All the images are loaded onto the machine from the service node via the service (JTAG) network.
 
  
−
; CNS
+
All the images are loaded onto the machine from the service node via the service (JTAG) network.  The loader is loaded first, followed by the CN- and ION-specific images loaded in order.
−
: Common Node Services, a proprietary "firmware" loaded onto all compute and I/O nodes, which negotiates between the hardware and the kernel.
 
−
; CNK
 
−
: Compute node kernel
 
−
; INK
 
−
: I/O node kernel
 
−
; ramdisk
 
−
: I/O Node ramdisk
 
  
−
All these files are loaded onto the machine from the service node via the service (JTAG) network.  <tt>uloader</tt> and <tt>CNS</tt> are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
+
The CN image list defaults to the CNS followed by the CNK.  The ION image list defaults to the CNS, followed by the Linux kernel, followed by the Linux ramdisk.  CNS stands for Common Node Services &#8211; it is a proprietary "firmware" which negotiates between the hardware and the kernel.
  
−
The system allows us to specify multiple CN images. Command Node Service(CNS)
+
To enable ZeptoOS, we need to boot the ZeptoOS CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
−
image and CN kernel image(IBM CNK) are default images and loaded into CN's main memory in order.  
 
−
/bgsys/drivers/ppcfloor/boot/cns is only choice for CNS, which is not an open source code , either.
 
−
We can also specify multiple ION images. CNS, IBM ION Linux kernel image and ION Linux ramdisk are default images. They are also loaded into ION's main memory in order.  
 
  
−
To enable Zepto feature, we need to boot Zepto CN kernel and ION kernel in a partition that we use.
+
In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.
−
We describe how to assign and boot Zepto images in this section.
 
  
−
===Cobalt installed system===
+
===System using Cobalt===
−
 
−
If your BGP system has the cobalt scheduler installed and its kernel
 
−
profile feature has been configured properly, it would be easy to
 
−
boot Zepto kernel for your task.
 
 
   
 
   
−
What you need is to make a directory in the kernel profile directory
+
If the BGP system in question has the [http://trac.mcs.anl.gov/projects/cobalt/ Cobalt] scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS should be easy.  All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.
−
and create a couple of symbolic link that point to Zepto images.
 
  
−
In ANL BGP, /bgsys/argonne-utils/profiles/ is the kernel profile directory.
+
On Argonne BGP machines, <tt>/bgsys/argonne-utils/profiles/</tt> is the top-level kernel profile directory; individual profiles are stored in its subdirectories.  Assuming that the ZeptoOS images have already been created (see [[Configuration]]), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:
−
Here are the concrete steps to create a new kernel profile.  Suppose that  
 
−
you have already built your Zepto kernel images and write permission to the kernel profile directory.
 
 
   
 
   
−
<pre>$ cd KERNEL_PROFILE_DIR
+
<pre>
−
$ mkdir YOUR_PROFILE_NAME && cd YOUR_PROFILE_NAME
+
$ cd <kernel_profile_dir>
−
$ ln -s ZEPTO_DIR/BGP-CN-zImage-with-initrd.elf CNK  
+
$ mkdir <your_profile> && cd <your_profile>
−
$ ln -s ZEPTO_DIR/BGP-ION-zImage.elf INK  
+
$ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK
−
$ ln -s ZEPTO_DIR/BGP-ION-ramdisk-for-CNL.elf ramdisk  
+
$ ln -s <zepto_dir>/BGP-ION-zImage.elf INK
−
$ ln -s ../factory-default/CNS  
+
$ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk
−
$ ln -s ../factory-default/uloader  
+
$ ln -s ../default/CNS
 +
$ ln -s ../default/uloader
 
</pre>  
 
</pre>  
−
+
 
−
NOTE: your Zepto images must be readable from others, otherwise your
+
'''Note:''' ensure that the ZeptoOS images are world-readable from the service node, otherwise jobs will fail to start.  If needed, copy the images to the kernel profile rather than link to them.
−
job will fail. Please double check!!!
+
 
−
+
For Argonne users, we provide a convenience script <tt>mkprofile-ANL.sh</tt> with some extra features. The following command line is equivalent to the above steps:
−
For ANL user, we provide a convenient script named mkprofile-ANL.sh  
+
 
−
which essentially does what mentioned in above but has some extra  
+
<pre>
−
features. The following command line is equivalent to the steps  
+
$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>
−
described in above.
 
−
 
−
<pre>$ cd ZEPTO_DIR && ./mkprofile-ANL.sh --profile=YOUR_PROFILE_NAME
 
 
</pre>  
 
</pre>  
 
   
 
   
−
Invoking it with the -h option shows its help message. Use -c if you  
+
Invoking the script with the <tt>-h</tt> option prints an overview of command line options. In particular, use <tt>-c</tt> if you prefer to copy images instead of making the links.
−
actually need to copy images instead of making symbolic link.  Use
 
−
-cn, -ion or -rd if you have a custom named image.  
 
 
   
 
   
−
<pre>$ ./mkprofile-ANL.sh -h  
+
<pre>
−
Usage: ./mkprofile-ANL.sh [OPTIONS]  
+
$ ./mkprofile-ANL.sh -h
 +
Usage: ./mkprofile-ANL.sh [OPTIONS]
 +
 
 +
Options:
 +
-h            : Show this message
 +
-c            : Copy images instead of making symbolic link
 +
-f            : Overwrite existing profile
 +
--profile=name : Specify profile name
 +
--cn=fn        : Compute Node Kernel Image
 +
--ion=fn      : Specify I/O Node Kernel Image
 +
--rd=fn        : Specify I/O Node Ramdisk Image
 +
--ls          : show files in profile
 +
--dryrun
 +
</pre>
 +
 
 +
Once the ZeptoOS kernel profile is set up, it can be booted
 +
by specifying the profile name when submitting a job using either <tt>cqsub</tt> or <tt>qsub</tt>:
 
   
 
   
−
Options:
+
<pre>
−
-h            : Show this message
+
$ cqsub -k <profile_name> ...
−
-c            : Copy images instead of making symbolic link
+
$ qsub --kernel <profile_name> ...
−
-f            : Overwrite existing profile 
 
−
--profile=name : Specify profile name
 
−
--cn=fn        : Compute Node Kernel Image 
 
−
--ion=fn      : Specify I/O Node Kernel Image     
 
−
--rd=fn        : Specify I/O Node Ramdisk Image
 
−
--ls          : show files in profile
 
−
--dryrun 
 
−
</pre>  
 
−
 
−
Once you have properly setup your Zepto kernel profile, you can
 
−
boot Zepto kernel by specifying your kernel profile name via the -k  
 
−
cobalt option.  
 
−
 
−
<pre>$ cqsub -k YOUR_PROFILE_NAME ....  
 
 
</pre>
 
</pre>
  
−
===MMCS console===
+
Testing and troubleshooting ZeptoOS is discussed in detail in the [[Testing|next section]].
 +
 
 +
===Manual installation using the MMCS console===
  
−
Cobalt is an open source software. No guarantee that your system has colbalt scheduler installed.
+
If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS). MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.
−
If you don't have cobalt, probably we'll need to use Midplane Management Control System(MMCS).  
 
−
We explain how to assign and boot your own kernel images using MMCS.
 
  
−
Here is a brief summary of MMCS.
+
Before we begin, please select a partition to experiment on. The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).
−
* Available on all BGP systems
 
−
* the lowest control mechanism for BGP partition
 
−
* allocate, free or query of block(partition)
 
−
* status check
 
−
* assign boot images
 
−
* low level debug command
 
  
−
Due to its low level interface, it requires administrator level permission to use it.
+
====Assigning ZeptoOS images to the partition====
−
You also need to reserve a partition (or block).
 
  
 +
Log on the service node and start the MMCS console:
  
−
====Assign Zepto images to a BGP partition====
+
<pre>
 +
$ ssh <service_node>
 +
sn $ mmcs_db_console
 +
connecting to mmcs_server
 +
connected to mmcs_server
 +
connected to DB2
 +
mmcs$
 +
</pre>
 +
 
 +
We begin by obtaining the current boot configuration of the partition, so that at the end we can revert it to the initial condition:
 +
 
 +
<pre>
 +
mmcs$ getblockinfo <partition_name>
 +
OK
 +
boot info for block <partition_name>:
 +
mloader: /bgsys/drivers/ppcfloor/boot/uloader
 +
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk
 +
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
 +
status: F
 +
mmcs$
 +
</pre>
 +
 
 +
Now assign ZeptoOS images to the partition:
 +
 
 +
<pre>
 +
mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
 +
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-CN-zImage-with-initrd.elf \
 +
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf
 +
mmcs$ quit
 +
</pre>
 +
 
 +
====Booting ZeptoOS====
 
   
 
   
−
Login to the service node and start MMCS.
+
Once the partition has been correctly configured, the ZeptoOS images will be loaded when you run a job on that partition (via <tt>mpirun</tt>, for example):
 
   
 
   
−
<pre>$ ssh sn       
+
<pre>
−
sn $ ./mmcs.sh 
+
$ mpirun -verbose 1 -partition <partition_name> -np 64 -timeout 600 -cwd $PWD -exe ./a.out
−
</pre>  
+
</pre>
−
+
 
−
<pre>[mmcs.sh]
+
Testing and troubleshooting ZeptoOS is discussed in detail in the [[Testing|next section]].
−
#!/bin/sh
+
 
−
+
====Restore the original configuration====
−
export DB2HOME=/dbhome/bgpdb2c/sqllib
+
 
−
DB2SRC=${DB2HOME}/db2profile
+
'''Do not forget this step!'''
−
[ -f "$DB2SRC" ] && . $DB2SRC
+
 
−
+
Once the experiments are over, restore the partition to its original configuration:
−
cd /bgsys/drivers/ppcfloor/bin
 
−
./mmcs_db_console
 
−
</pre>
 
−
 
−
Please memorize that the current configuration. You need to revert the  
 
−
blockinfo to the original configuration later on after you are done with Zepto kernel.
 
−
 
−
<pre>console $ set_username YOUR_LOGIN_NAME
 
−
console $ getblockinfo BGP_BLOCK_NAME
 
−
OK
 
−
boot info for block BGP_BLOCK_NAME:  
 
−
mloader: /bgsys/drivers/ppcfloor/boot/uloader
 
−
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk
 
−
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
 
−
status: F
 
−
</pre>
 
−
 
−
Assign Zepto images to a parition
 
−
 
−
<pre>console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,BGP_CN_LINUX_KERNEL_PATH /bgsys/drivers/ppcfloor\
 
−
/boot/cns,BGP_ION_LINUX_KERNEL_PATH,BGP_ION_LINUX_RAMDISK_PATH
 
−
console $ quit
 
−
</pre>
 
  
−
====Boot Zepto kernel====
+
<pre>
−
+
$ ssh <service_node>
−
Once you have configured a partition with Zepto kernels correctly,
+
sn $ mmcs_db_console
−
Zepto kernels will be booted when you run a job on that partition(via
+
connecting to mmcs_server
−
mpirun for example)
+
connected to mmcs_server
−
+
connected to DB2
−
<pre>fen $ mpirun -verbose 1 -partition BGP_BLOCK_NAME  -np 64 -timeout 600 -cwd `pwd` -exe ./a.out
+
mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
−
</pre>  
+
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk \
−
+
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
−
====Restore to the original configuration(Don't forget!!!)====
+
mmcs$ quit
−
 
−
After you have done your work on Zepto kernel, you need to restore to
 
−
the original configuration. Here is an example.
 
−
 
−
<pre>fen $ ssh sn
 
−
sn $ ./mmcs.sh
 
−
console $ set_username YOUR_LOGIN_NAME
 
−
console $ setblockinfo BGP_BLOCK_NAME /bgsys/drivers/ppcfloor/boot/uloader /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk /bgsys/drivers/\
 
−
ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk  
 
−
console $ quit  
 
 
</pre>
 
</pre>
  
 
----
 
----
 
[[Configuration]] | [[ZeptoOS_Documentation|Top]] | [[Testing]]
 
[[Configuration]] | [[ZeptoOS_Documentation|Top]] | [[Testing]]

Latest revision as of 10:32, 8 May 2009

Configuration | Top | Testing

Installing the support files

Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) used primarily when building compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and type: 

$ python install.py /path/to/install

This will install the following files: 

Install Home:  /path/to/install
Creating install directories ...
Making /path/to/install/bin
Making /path/to/install/cnbin
Making /path/to/install/include
Making /path/to/install/lib
Making /path/to/install/lib/zoid

Installing MPICH2 scripts ...
/path/to/install/bin/zmpicc
/path/to/install/bin/zmpicxx
/path/to/install/bin/zmpif77
/path/to/install/bin/zmpif90
/path/to/install/bin/zmpixlc
/path/to/install/bin/zmpixlc_r
/path/to/install/bin/zmpixlcxx
/path/to/install/bin/zmpixlcxx_r
/path/to/install/bin/zmpixlf2003
/path/to/install/bin/zmpixlf2003_r
/path/to/install/bin/zmpixlf77
/path/to/install/bin/zmpixlf77_r
/path/to/install/bin/zmpixlf90
/path/to/install/bin/zmpixlf90_r
/path/to/install/bin/zmpixlf95
/path/to/install/bin/zmpixlf95_r

Installing Compute Node binaries ...
/path/to/install/cnbin/cn-ipfwd.sh
/path/to/install/cnbin/cn-ipfwd

Installing misc files ...
/path/to/install/bin/zelftool
/path/to/install/bin/zkparam.py

Installing MPICH2 headers ...
/path/to/install/include/mpe_thread.h
/path/to/install/include/mpi.h
/path/to/install/include/mpi.mod
/path/to/install/include/mpi_base.mod
/path/to/install/include/mpi_constants.mod
/path/to/install/include/mpi_sizeofs.mod
/path/to/install/include/mpicxx.h
/path/to/install/include/mpif.h
/path/to/install/include/mpio.h
/path/to/install/include/mpiof.h
/path/to/install/include/mpix.h

Installing MPICH2 libraries ...
/path/to/install/lib/libcxxmpich.zcl.a
/path/to/install/lib/libfmpich.zcl.a
/path/to/install/lib/libfmpich_.zcl.a
/path/to/install/lib/libmpich.zcl.a
/path/to/install/lib/libmpich.zclf90.a

Installing SPI libraries ...
/path/to/install/lib/libSPI.zcl.a
/path/to/install/lib/libzcl.a

Installing ZOID files ...
/path/to/install/lib/libzoid_cn.a
/path/to/install/lib/zoid/libc.a
/path/to/install/lib/zoid/libpthread.a

Installing DCMF files ...
/path/to/install/include/dcmf.h
/path/to/install/include/dcmf_collectives.h
/path/to/install/include/dcmf_coremath.h
/path/to/install/include/dcmf_globalcollectives.h
/path/to/install/include/dcmf_multisend.h
/path/to/install/include/dcmf_optimath.h
/path/to/install/lib/libdcmf.zcl.a
/path/to/install/lib/libdcmfcoll.zcl.a

Setting up a kernel profile

The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.

Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units. More importantly, the system supports using different software stacks (what we call kernel profiles) on different partitions at the same time. For our immediate purposes, this means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.

A kernel profile consists of the following elements:

loader
A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
CN image list
A series of images loaded onto each compute node.
ION image list
A series of images loaded onto each I/O node.

All the images are loaded onto the machine from the service node via the service (JTAG) network. The loader is loaded first, followed by the CN- and ION-specific images loaded in order.

The CN image list defaults to the CNS followed by the CNK. The ION image list defaults to the CNS, followed by the Linux kernel, followed by the Linux ramdisk. CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.

To enable ZeptoOS, we need to boot the ZeptoOS CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.

In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.

System using Cobalt

If the BGP system in question has the Cobalt scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS should be easy. All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.

On Argonne BGP machines, /bgsys/argonne-utils/profiles/ is the top-level kernel profile directory; individual profiles are stored in its subdirectories. Assuming that the ZeptoOS images have already been created (see Configuration), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile: 

$ cd <kernel_profile_dir>
$ mkdir <your_profile> && cd <your_profile>
$ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK
$ ln -s <zepto_dir>/BGP-ION-zImage.elf INK
$ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk
$ ln -s ../default/CNS
$ ln -s ../default/uloader

Note: ensure that the ZeptoOS images are world-readable from the service node, otherwise jobs will fail to start. If needed, copy the images to the kernel profile rather than link to them.

For Argonne users, we provide a convenience script mkprofile-ANL.sh with some extra features. The following command line is equivalent to the above steps: 

$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>

Invoking the script with the -h option prints an overview of command line options. In particular, use -c if you prefer to copy images instead of making the links. 

$ ./mkprofile-ANL.sh -h
Usage: ./mkprofile-ANL.sh [OPTIONS]

Options:
-h             : Show this message
-c             : Copy images instead of making symbolic link
-f             : Overwrite existing profile
--profile=name : Specify profile name
--cn=fn        : Compute Node Kernel Image
--ion=fn       : Specify I/O Node Kernel Image
--rd=fn        : Specify I/O Node Ramdisk Image
--ls           : show files in profile
--dryrun

Once the ZeptoOS kernel profile is set up, it can be booted by specifying the profile name when submitting a job using either cqsub or qsub: 

$ cqsub -k <profile_name> ...
$ qsub --kernel <profile_name> ...

Testing and troubleshooting ZeptoOS is discussed in detail in the next section.

Manual installation using the MMCS console

If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS). MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.

Before we begin, please select a partition to experiment on. The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).

Assigning ZeptoOS images to the partition

Log on the service node and start the MMCS console: 

$ ssh <service_node>
sn $ mmcs_db_console
connecting to mmcs_server
connected to mmcs_server
connected to DB2
mmcs$

We begin by obtaining the current boot configuration of the partition, so that at the end we can revert it to the initial condition: 

mmcs$ getblockinfo <partition_name>
OK
boot info for block <partition_name>:
mloader: /bgsys/drivers/ppcfloor/boot/uloader
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
status: F
mmcs$

Now assign ZeptoOS images to the partition: 

mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-CN-zImage-with-initrd.elf \
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf
mmcs$ quit

Booting ZeptoOS

Once the partition has been correctly configured, the ZeptoOS images will be loaded when you run a job on that partition (via mpirun, for example): 

$ mpirun -verbose 1 -partition <partition_name> -np 64 -timeout 600 -cwd $PWD -exe ./a.out

Testing and troubleshooting ZeptoOS is discussed in detail in the next section.

Restore the original configuration

Do not forget this step!

Once the experiments are over, restore the partition to its original configuration: 

$ ssh <service_node>
sn $ mmcs_db_console
connecting to mmcs_server
connected to mmcs_server
connected to DB2
mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk \
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
mmcs$ quit

Configuration | Top | Testing

Retrieved from "http:/// /zeptoos/index.php?title=Installation&oldid=593"