Difference between revisions of "Installation"

From ZeptoOS
Jump to navigationJump to search
 
(2 intermediate revisions by the same user not shown)
Line 4: Line 4:
 
==Installing the support files==
 
==Installing the support files==
  
−
Installing ZeptoOS consists of two parts.  In the first part we install support files (header files, libraries, scripts, etc) use primarily when building compute node binaries for ZeptoOS.  To perform this step, change to the top-level directory and type:
+
Installing ZeptoOS consists of two parts.  In the first part we install support files (header files, libraries, scripts, etc) used primarily when building compute node binaries for ZeptoOS.  To perform this step, change to the top-level directory and type:
 
<pre>
 
<pre>
 
$ python install.py /path/to/install
 
$ python install.py /path/to/install
Line 14: Line 14:
 
Creating install directories ...
 
Creating install directories ...
 
Making /path/to/install/bin
 
Making /path/to/install/bin
 +
Making /path/to/install/cnbin
 
Making /path/to/install/include
 
Making /path/to/install/include
 
Making /path/to/install/lib
 
Making /path/to/install/lib
 +
Making /path/to/install/lib/zoid
  
−
Installing mpich2 scripts ...
+
Installing MPICH2 scripts ...
−
Installing /path/to/install/bin/zmpicc
+
/path/to/install/bin/zmpicc
−
Installing /path/to/install/bin/zmpicxx
+
/path/to/install/bin/zmpicxx
−
Installing /path/to/install/bin/zmpif77
+
/path/to/install/bin/zmpif77
−
Installing /path/to/install/bin/zmpif90
+
/path/to/install/bin/zmpif90
−
Installing /path/to/install/bin/zmpixlc_r
+
/path/to/install/bin/zmpixlc
−
Installing /path/to/install/bin/zmpixlc
+
/path/to/install/bin/zmpixlc_r
−
Installing /path/to/install/bin/zmpixlcxx_r
+
/path/to/install/bin/zmpixlcxx
−
Installing /path/to/install/bin/zmpixlcxx
+
/path/to/install/bin/zmpixlcxx_r
−
Installing /path/to/install/bin/zmpixlf2003_r
+
/path/to/install/bin/zmpixlf2003
−
Installing /path/to/install/bin/zmpixlf2003
+
/path/to/install/bin/zmpixlf2003_r
−
Installing /path/to/install/bin/zmpixlf77_r
+
/path/to/install/bin/zmpixlf77
−
Installing /path/to/install/bin/zmpixlf77
+
/path/to/install/bin/zmpixlf77_r
−
Installing /path/to/install/bin/zmpixlf90_r
+
/path/to/install/bin/zmpixlf90
−
Installing /path/to/install/bin/zmpixlf90
+
/path/to/install/bin/zmpixlf90_r
−
Installing /path/to/install/bin/zmpixlf95_r
+
/path/to/install/bin/zmpixlf95
−
Installing /path/to/install/bin/zmpixlf95
+
/path/to/install/bin/zmpixlf95_r
 +
 
 +
Installing Compute Node binaries ...
 +
/path/to/install/cnbin/cn-ipfwd.sh
 +
/path/to/install/cnbin/cn-ipfwd
  
 
Installing misc files ...
 
Installing misc files ...
 
/path/to/install/bin/zelftool
 
/path/to/install/bin/zelftool
 +
/path/to/install/bin/zkparam.py
  
 
Installing MPICH2 headers ...
 
Installing MPICH2 headers ...
 
/path/to/install/include/mpe_thread.h
 
/path/to/install/include/mpe_thread.h
 +
/path/to/install/include/mpi.h
 +
/path/to/install/include/mpi.mod
 
/path/to/install/include/mpi_base.mod
 
/path/to/install/include/mpi_base.mod
 
/path/to/install/include/mpi_constants.mod
 
/path/to/install/include/mpi_constants.mod
 +
/path/to/install/include/mpi_sizeofs.mod
 
/path/to/install/include/mpicxx.h
 
/path/to/install/include/mpicxx.h
 
/path/to/install/include/mpif.h
 
/path/to/install/include/mpif.h
−
/path/to/install/include/mpi.h
+
/path/to/install/include/mpio.h
−
/path/to/install/include/mpi.mod
 
 
/path/to/install/include/mpiof.h
 
/path/to/install/include/mpiof.h
−
/path/to/install/include/mpio.h
 
−
/path/to/install/include/mpi_sizeofs.mod
 
 
/path/to/install/include/mpix.h
 
/path/to/install/include/mpix.h
  
 
Installing MPICH2 libraries ...
 
Installing MPICH2 libraries ...
 
/path/to/install/lib/libcxxmpich.zcl.a
 
/path/to/install/lib/libcxxmpich.zcl.a
 +
/path/to/install/lib/libfmpich.zcl.a
 
/path/to/install/lib/libfmpich_.zcl.a
 
/path/to/install/lib/libfmpich_.zcl.a
−
/path/to/install/lib/libfmpich.zcl.a
 
 
/path/to/install/lib/libmpich.zcl.a
 
/path/to/install/lib/libmpich.zcl.a
 
/path/to/install/lib/libmpich.zclf90.a
 
/path/to/install/lib/libmpich.zclf90.a
Line 62: Line 69:
 
/path/to/install/lib/libzcl.a
 
/path/to/install/lib/libzcl.a
  
−
Installing Zoid files ...
+
Installing ZOID files ...
 
/path/to/install/lib/libzoid_cn.a
 
/path/to/install/lib/libzoid_cn.a
 +
/path/to/install/lib/zoid/libc.a
 +
/path/to/install/lib/zoid/libpthread.a
  
 
Installing DCMF files ...
 
Installing DCMF files ...
 +
/path/to/install/include/dcmf.h
 
/path/to/install/include/dcmf_collectives.h
 
/path/to/install/include/dcmf_collectives.h
 
/path/to/install/include/dcmf_coremath.h
 
/path/to/install/include/dcmf_coremath.h
 
/path/to/install/include/dcmf_globalcollectives.h
 
/path/to/install/include/dcmf_globalcollectives.h
−
/path/to/install/include/dcmf.h
 
 
/path/to/install/include/dcmf_multisend.h
 
/path/to/install/include/dcmf_multisend.h
 
/path/to/install/include/dcmf_optimath.h
 
/path/to/install/include/dcmf_optimath.h
 +
/path/to/install/lib/libdcmf.zcl.a
 
/path/to/install/lib/libdcmfcoll.zcl.a
 
/path/to/install/lib/libdcmfcoll.zcl.a
−
/path/to/install/lib/libdcmf.zcl.a
 
 
</pre>
 
</pre>
  
Line 80: Line 89:
 
The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.
 
The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.
  
−
Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units.  More importantly, the system supports using different software stacks (what we call ''kernel profiles'') on different partitions at the same time.  This means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.
+
Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units.  More importantly, the system supports using different software stacks (what we call ''kernel profiles'') on different partitions at the same time.  For our immediate purposes, this means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.
  
 
A kernel profile consists of the following elements:
 
A kernel profile consists of the following elements:
Line 93: Line 102:
 
All the images are loaded onto the machine from the service node via the service (JTAG) network.  The loader is loaded first, followed by the CN- and ION-specific images loaded in order.
 
All the images are loaded onto the machine from the service node via the service (JTAG) network.  The loader is loaded first, followed by the CN- and ION-specific images loaded in order.
  
−
The CN image list defaults to CNS followed by the CNK.  The ION image list defaults to CNS, followed by the Linux kernel, followed by the Linux ramdisk.  CNS stands for Common Node Services &#8211; it is a proprietary "firmware" which negotiates between the hardware and the kernel.
+
The CN image list defaults to the CNS followed by the CNK.  The ION image list defaults to the CNS, followed by the Linux kernel, followed by the Linux ramdisk.  CNS stands for Common Node Services &#8211; it is a proprietary "firmware" which negotiates between the hardware and the kernel.
  
−
To enable ZeptoOS, we need to boot the Zepto CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use.  The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
+
To enable ZeptoOS, we need to boot the ZeptoOS CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use.  The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.
  
 
In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.
 
In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.
Line 101: Line 110:
 
===System using Cobalt===
 
===System using Cobalt===
 
   
 
   
−
If the BGP system in question has the [http://trac.mcs.anl.gov/projects/cobalt/ Cobalt] scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS there should be easy.  All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.
+
If the BGP system in question has the [http://trac.mcs.anl.gov/projects/cobalt/ Cobalt] scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS should be easy.  All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.
  
 
On Argonne BGP machines, <tt>/bgsys/argonne-utils/profiles/</tt> is the top-level kernel profile directory; individual profiles are stored in its subdirectories.  Assuming that the ZeptoOS images have already been created (see [[Configuration]]), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:
 
On Argonne BGP machines, <tt>/bgsys/argonne-utils/profiles/</tt> is the top-level kernel profile directory; individual profiles are stored in its subdirectories.  Assuming that the ZeptoOS images have already been created (see [[Configuration]]), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile:
Line 115: Line 124:
 
</pre>  
 
</pre>  
  
−
'''Note:''' ensure that the ZeptoOS images are world-readable, otherwise jobs will fail to start.  If needed, copy the images to the kernel profile rather than link to them.
+
'''Note:''' ensure that the ZeptoOS images are world-readable from the service node, otherwise jobs will fail to start.  If needed, copy the images to the kernel profile rather than link to them.
  
 
For Argonne users, we provide a convenience script <tt>mkprofile-ANL.sh</tt> with some extra features. The following command line is equivalent to the above steps:
 
For Argonne users, we provide a convenience script <tt>mkprofile-ANL.sh</tt> with some extra features. The following command line is equivalent to the above steps:
Line 123: Line 132:
 
</pre>  
 
</pre>  
 
   
 
   
−
Invoking it with the <tt>-h</tt> option prints an overview of command line options. In particular, use <tt>-c</tt> if you prefer to copy images instead of making the links.
+
Invoking the script with the <tt>-h</tt> option prints an overview of command line options. In particular, use <tt>-c</tt> if you prefer to copy images instead of making the links.
 
   
 
   
 
<pre>
 
<pre>
Line 155: Line 164:
 
If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS).  MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.
 
If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS).  MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.
  
−
Before we begin, please select a partition that we can experiment on.  The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).
+
Before we begin, please select a partition to experiment on.  The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).
  
 
====Assigning ZeptoOS images to the partition====
 
====Assigning ZeptoOS images to the partition====
Line 170: Line 179:
 
</pre>
 
</pre>
  
−
We start by obtaining the current configuration of a partition, so that at the end we can revert it to the initial condition:
+
We begin by obtaining the current boot configuration of the partition, so that at the end we can revert it to the initial condition:
  
 
<pre>
 
<pre>
Line 190: Line 199:
 
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf
 
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf
 
mmcs$ quit
 
mmcs$ quit
−
</pre>  
+
</pre>
  
 
====Booting ZeptoOS====
 
====Booting ZeptoOS====
 
   
 
   
−
Once you have correctly configured a partition, the ZeptoOS images will be loaded when you run a job on that partition (via <tt>mpirun</tt>, for example):
+
Once the partition has been correctly configured, the ZeptoOS images will be loaded when you run a job on that partition (via <tt>mpirun</tt>, for example):
 
   
 
   
 
<pre>
 
<pre>
Line 206: Line 215:
 
'''Do not forget this step!'''
 
'''Do not forget this step!'''
  
−
After you are finished using ZeptoOS, you need to restore the partition to its original configuration:
+
Once the experiments are over, restore the partition to its original configuration:
  
 
<pre>
 
<pre>

Latest revision as of 09:32, 8 May 2009

Configuration | Top | Testing

Installing the support files

Installing ZeptoOS consists of two parts. In the first part we install support files (header files, libraries, scripts, etc) used primarily when building compute node binaries for ZeptoOS. To perform this step, change to the top-level directory and type: 

$ python install.py /path/to/install

This will install the following files: 

Install Home:  /path/to/install
Creating install directories ...
Making /path/to/install/bin
Making /path/to/install/cnbin
Making /path/to/install/include
Making /path/to/install/lib
Making /path/to/install/lib/zoid

Installing MPICH2 scripts ...
/path/to/install/bin/zmpicc
/path/to/install/bin/zmpicxx
/path/to/install/bin/zmpif77
/path/to/install/bin/zmpif90
/path/to/install/bin/zmpixlc
/path/to/install/bin/zmpixlc_r
/path/to/install/bin/zmpixlcxx
/path/to/install/bin/zmpixlcxx_r
/path/to/install/bin/zmpixlf2003
/path/to/install/bin/zmpixlf2003_r
/path/to/install/bin/zmpixlf77
/path/to/install/bin/zmpixlf77_r
/path/to/install/bin/zmpixlf90
/path/to/install/bin/zmpixlf90_r
/path/to/install/bin/zmpixlf95
/path/to/install/bin/zmpixlf95_r

Installing Compute Node binaries ...
/path/to/install/cnbin/cn-ipfwd.sh
/path/to/install/cnbin/cn-ipfwd

Installing misc files ...
/path/to/install/bin/zelftool
/path/to/install/bin/zkparam.py

Installing MPICH2 headers ...
/path/to/install/include/mpe_thread.h
/path/to/install/include/mpi.h
/path/to/install/include/mpi.mod
/path/to/install/include/mpi_base.mod
/path/to/install/include/mpi_constants.mod
/path/to/install/include/mpi_sizeofs.mod
/path/to/install/include/mpicxx.h
/path/to/install/include/mpif.h
/path/to/install/include/mpio.h
/path/to/install/include/mpiof.h
/path/to/install/include/mpix.h

Installing MPICH2 libraries ...
/path/to/install/lib/libcxxmpich.zcl.a
/path/to/install/lib/libfmpich.zcl.a
/path/to/install/lib/libfmpich_.zcl.a
/path/to/install/lib/libmpich.zcl.a
/path/to/install/lib/libmpich.zclf90.a

Installing SPI libraries ...
/path/to/install/lib/libSPI.zcl.a
/path/to/install/lib/libzcl.a

Installing ZOID files ...
/path/to/install/lib/libzoid_cn.a
/path/to/install/lib/zoid/libc.a
/path/to/install/lib/zoid/libpthread.a

Installing DCMF files ...
/path/to/install/include/dcmf.h
/path/to/install/include/dcmf_collectives.h
/path/to/install/include/dcmf_coremath.h
/path/to/install/include/dcmf_globalcollectives.h
/path/to/install/include/dcmf_multisend.h
/path/to/install/include/dcmf_optimath.h
/path/to/install/lib/libdcmf.zcl.a
/path/to/install/lib/libdcmfcoll.zcl.a

Setting up a kernel profile

The second part of the ZeptoOS installation is the process of setting up a ZeptoOS kernel profile.

Blue Gene systems are partitionable, meaning that the hardware can be split into multiple, largely independent, sub-units. More importantly, the system supports using different software stacks (what we call kernel profiles) on different partitions at the same time. For our immediate purposes, this means that one can safely experiment with ZeptoOS on one partition while other users are running production jobs using the default light-weight kernel on the rest of the machine.

A kernel profile consists of the following elements:

loader
A proprietary low-level bootstrap code, loaded onto all compute and I/O nodes.
CN image list
A series of images loaded onto each compute node.
ION image list
A series of images loaded onto each I/O node.

All the images are loaded onto the machine from the service node via the service (JTAG) network. The loader is loaded first, followed by the CN- and ION-specific images loaded in order.

The CN image list defaults to the CNS followed by the CNK. The ION image list defaults to the CNS, followed by the Linux kernel, followed by the Linux ramdisk. CNS stands for Common Node Services – it is a proprietary "firmware" which negotiates between the hardware and the kernel.

To enable ZeptoOS, we need to boot the ZeptoOS CN kernel and ramdisk and the Zepto ION kernel and ramdisk in a partition that we want to use. The loader and the CNS are closed-source components, so ZeptoOS uses the same images for them as the default kernel profile.

In the remainder of this section we discuss how to assign and boot ZeptoOS-specific images.

System using Cobalt

If the BGP system in question has the Cobalt scheduler installed and its kernel profile feature has been configured properly, then using ZeptoOS should be easy. All that is necessary is to make a subdirectory in the kernel profile directory and to create a couple of symbolic links that point to ZeptoOS images.

On Argonne BGP machines, /bgsys/argonne-utils/profiles/ is the top-level kernel profile directory; individual profiles are stored in its subdirectories. Assuming that the ZeptoOS images have already been created (see Configuration), and that one has write permissions to the kernel profile directory, here are the steps to create a new kernel profile: 

$ cd <kernel_profile_dir>
$ mkdir <your_profile> && cd <your_profile>
$ ln -s <zepto_dir>/BGP-CN-zImage-with-initrd.elf CNK
$ ln -s <zepto_dir>/BGP-ION-zImage.elf INK
$ ln -s <zepto_dir>/BGP-ION-ramdisk-for-CNL.elf ramdisk
$ ln -s ../default/CNS
$ ln -s ../default/uloader

Note: ensure that the ZeptoOS images are world-readable from the service node, otherwise jobs will fail to start. If needed, copy the images to the kernel profile rather than link to them.

For Argonne users, we provide a convenience script mkprofile-ANL.sh with some extra features. The following command line is equivalent to the above steps: 

$ cd <zepto_dir> && ./mkprofile-ANL.sh --profile=<your_profile>

Invoking the script with the -h option prints an overview of command line options. In particular, use -c if you prefer to copy images instead of making the links. 

$ ./mkprofile-ANL.sh -h
Usage: ./mkprofile-ANL.sh [OPTIONS]

Options:
-h             : Show this message
-c             : Copy images instead of making symbolic link
-f             : Overwrite existing profile
--profile=name : Specify profile name
--cn=fn        : Compute Node Kernel Image
--ion=fn       : Specify I/O Node Kernel Image
--rd=fn        : Specify I/O Node Ramdisk Image
--ls           : show files in profile
--dryrun

Once the ZeptoOS kernel profile is set up, it can be booted by specifying the profile name when submitting a job using either cqsub or qsub: 

$ cqsub -k <profile_name> ...
$ qsub --kernel <profile_name> ...

Testing and troubleshooting ZeptoOS is discussed in detail in the next section.

Manual installation using the MMCS console

If the system does not use Cobalt, we will need to resort to using Blue Gene Midplane Management Control System (MMCS). MMCS is a low-level control mechanism; administrator-level access permissions are required to use it.

Before we begin, please select a partition to experiment on. The partition should be unused; if some reservation system is in place, please reserve that partition (but do not boot it yet).

Assigning ZeptoOS images to the partition

Log on the service node and start the MMCS console: 

$ ssh <service_node>
sn $ mmcs_db_console
connecting to mmcs_server
connected to mmcs_server
connected to DB2
mmcs$

We begin by obtaining the current boot configuration of the partition, so that at the end we can revert it to the initial condition: 

mmcs$ getblockinfo <partition_name>
OK
boot info for block <partition_name>:
mloader: /bgsys/drivers/ppcfloor/boot/uloader
cnloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk
ioloadImg: /bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
status: F
mmcs$

Now assign ZeptoOS images to the partition: 

mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-CN-zImage-with-initrd.elf \
/bgsys/drivers/ppcfloor/boot/cns,<zepto_dir>/BGP-ION-zImage.elf,<zepto_dir>/BGP-ION-ramdisk-for-CNL.elf
mmcs$ quit

Booting ZeptoOS

Once the partition has been correctly configured, the ZeptoOS images will be loaded when you run a job on that partition (via mpirun, for example): 

$ mpirun -verbose 1 -partition <partition_name> -np 64 -timeout 600 -cwd $PWD -exe ./a.out

Testing and troubleshooting ZeptoOS is discussed in detail in the next section.

Restore the original configuration

Do not forget this step!

Once the experiments are over, restore the partition to its original configuration: 

$ ssh <service_node>
sn $ mmcs_db_console
connecting to mmcs_server
connected to mmcs_server
connected to DB2
mmcs$ setblockinfo <partition_name> /bgsys/drivers/ppcfloor/boot/uloader \
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/cnk \
/bgsys/drivers/ppcfloor/boot/cns,/bgsys/drivers/ppcfloor/boot/linux,/bgsys/drivers/ppcfloor/boot/ramdisk
mmcs$ quit

Configuration | Top | Testing

Retrieved from "http:/// /zeptoos/index.php?title=Installation&oldid=593"