Testing - Revision history

Iskra at 16:56, 8 May 2009

2009-05-08T16:56:52Z

Iskra at 15:33, 8 May 2009

2009-05-08T15:33:59Z

Iskra: /* Submitting */

2009-05-07T20:43:51Z

Submitting

Iskra: /* Interactive login */

2009-05-07T20:12:54Z

Interactive login

Iskra: /* Interactive login */

2009-05-07T20:01:25Z

Interactive login

Kazutomo: /* Compiling */

2009-05-06T19:44:51Z

Compiling

Iskra: /* The /bin/sleep job */

2009-05-06T16:01:38Z

The /bin/sleep job

Iskra at 15:54, 6 May 2009

2009-05-06T15:54:36Z

Iskra at 20:03, 5 May 2009

2009-05-05T20:03:02Z

Iskra: /* Shell script job */

2009-05-04T21:44:55Z

Shell script job

@@ Line 20: / Line 20: @@
 </pre>
-This test, if successful, will verify that the ZeptoOS compute and I/O node environments are booting correctly.  We deliberately chose a system binary such as <tt>/bin/sleep</tt> instead of something from a network filesystem so that even if the network filesystem does not come up for some reason, the test can still succeed.
+This test, if successful, will verify that the ZeptoOS compute and I/O node environments are booting correctly.  We deliberately chose a system binary such as <tt>/bin/sleep</tt> instead of something from a network file system to reduce the number of dependencies.
 If everything works out fine, messages such as the following will be found in the error stream (''jobid''.error file if using Cobalt):
@@ Line 61: / Line 61: @@
 </pre>
-This error indicates that the job was submitted to the default software environment, not to ZeptoOS (at the very least, the default I/O node ramdisk was used).  You need to go back to the [[Installation#Setting up a kernel profile|Installation]] section to fix the problem.  Information from the system log files can be useful to diagnose the problem.
+This error indicates that the job was submitted to the default software environment with the light-weight kernel, not to ZeptoOS (at the very least, the default I/O node ramdisk was used).  Go back to the [[Installation#Setting up a kernel profile|Installation]] section to fix the problem.  Information from the system log files (see below) can be useful to diagnose the problem.
 ==Log files==
@@ Line 131: / Line 131: @@
 (again, we stripped the prefixes to make the lines shorter)
-Messages such as <tt>Zepto ION startup</tt> or <tt>Starting ZOID</tt> clearly indicate that a ZeptoOS I/O node ramdisk is being used.  If one instead mistakenly booted with the default ramdisk, this could be recognized by messages such as:
+Messages such as <tt>Zepto ION startup</tt> or <tt>Starting ZOID</tt> clearly indicate that a ZeptoOS I/O node ramdisk is being used.  If instead one mistakenly boots with the default ramdisk, this could be recognized by messages such as:
 <pre>
@@ Line 138: / Line 138: @@
 </pre>
-(<tt>ciod</tt> is ''never'' started when using Zepto Compute Node Linux)
+(<tt>ciod</tt> is ''never'' started when using the ZeptoOS compute node Linux)
 In addition to verifying the ramdisk, the correct I/O node kernel can also be verified using the I/O node logfile by checking the kernel build timestamp in the first line of the boot log.  As of this writing the default kernel on the Argonne machines has a timestamp of <tt>Wed Oct 29 18:51:19 UTC 2008</tt>; as can be seen above, the ZeptoOS kernel was built more recently.
@@ Line 144: / Line 144: @@
 ===Compute node===
-All the compute nodes on the machine share the same MMCS log file, located in <tt>/bgsys/logs/BGP/</tt>.  The name of the log file is not fixed (it contains a timestamp), but <tt>sn1-bgdb0-mmcs_db_server-current.log</tt> always links to the current file.  Because the file is shared with other jobs, we recommed to grep it for user name, partition name, or both.
+All the compute nodes on the machine share the same MMCS log file, located in <tt>/bgsys/logs/BGP/</tt>.  The name of the log file is not fixed (it contains a timestamp), but <tt><service_node>-bgdb0-mmcs_db_server-current.log</tt> always links to the current file.  Because the file is shared with other jobs, we recommed to grep it for user name, partition name, or both.
-A correct boot log when when booting ZeptoOS will look something like this:
+A correct boot log when booting ZeptoOS will look something like this:
 <pre>
@@ Line 299: / Line 299: @@
 </pre>
-If everything is configured correctly, SSH will only let in root and the partition owner; no other unprivileged user will be allowed on the node.  However, this might require site-specific customization to work properly.  To enable access for the partition owner, one might need to make adjustments to [[ZOID#The /bin.rd/update_passwd_file.sh file|update_passwd_file.sh]].  To enable password-less login for the partition owners without requiring them to set up personal SSH keypairs, we recommend to add the names of the front end nodes to the <tt>shosts.equiv</tt> file, found in <tt>ramdisk/ION/ramdisk-add/etc/ssh.zepto/</tt> (it is empty by default; remember to use the names from the network that interconnects front end and I/O nodes, which might be different from hostnames, e.g., at Argonne we need to add the <tt>-data</tt> suffix to the hostnames).  Until this has all been set up, one might prefer to log on as root (<tt>ssh -l root</tt>), passing the password provided while [[Configuration#Building|building]] the ZeptoOS environment.
+If everything is configured correctly, SSH will only let in root and the partition owner; no other unprivileged user will be allowed on the node.  However, this might require site-specific customizations to work properly.  To enable access for the partition owner, an administrator might need to make adjustments to [[ZOID#The /bin.rd/update_passwd_file.sh file|update_passwd_file.sh]].  To enable password-less login for the partition owners without requiring them to set up personal SSH key pairs, we recommend to add the names of the front end nodes to the <tt>shosts.equiv</tt> file, found in <tt>ramdisk/ION/ramdisk-add/etc/ssh.zepto/</tt> (it is empty by default; remember to use the names from the network that interconnects front end and I/O nodes, which might be different from hostnames, e.g., at Argonne we need to add the <tt>-data</tt> suffix to the hostnames).  Until this has all been set up, one might prefer to log on as root (<tt>ssh -l root</tt>), passing the password provided when [[Configuration#Building|building]] the ZeptoOS environment.
 Also, even when the partition owner is correctly set up, there will be a time window while booting the I/O node when the SSH daemon is already running, but a job has not yet been started; during that window, the partition owner cannot log on. If that happens, wait a few seconds and try again.
-Here's part of the <tt>ps</tt> output from the I/O node:
+Here is part of the <tt>ps</tt> output from an I/O node:
 <pre>
@@ Line 337: / Line 337: @@
 </pre>
-The I/O nodes run a small Linux setup with the root filesystem in the ramdisk.  Custom processes can be started, just like on any ordinary Linux node.  In the example above, it is mostly a few system daemons and the remote filesystem clients (GPFS, PVFS).  Please verify at this stage that the remote filesystem have been mounted correctly.
+The I/O nodes run a small Linux setup with the root file system in the ramdisk.  Custom processes can be started, just like on any ordinary Linux node.  In the example above, it is mostly a few system daemons and the remote file system clients (GPFS, PVFS).  Please verify at this stage that the remote file systems have been mounted correctly.
-One custom process running on the node is [[ZOID]], the I/O forwarding and job control daemon, which enables the communication with the compute nodes.  One of the facilities offered by ZOID is IP forwarding between the I/O node and the compute nodes, implemented using the virtual network tunneling device available in Linux:
+One custom process running on the node is [[ZOID]], the I/O forwarding and job control daemon, which enables the communication with the compute nodes.  One of the facilities offered by ZOID is IP forwarding between the I/O nodes and the compute nodes, implemented using the virtual network tunneling device available in Linux:
 <pre>
@@ Line 353: / Line 353: @@
 </pre>
-At least on Argonne machines, with a 64:1 ratio of compute nodes to I/O nodes, compute nodes have addresses <tt>192.168.1.1</tt> to <tt>192.168.1.64</tt> (the last octet of the address is the [[FAQ#Pset rank|pset rank]]).  Somewhat confusingly, the first compute node (compute node <tt>0</tt>) has IP address <tt>192.168.1.64</tt>, so if one submits a one-node job as we did, that is the IP address that needs to be used to log on that sole running compute node.  The IP address of the second compute node is... <tt>192.168.1.59</tt>.  On a machine with a 16:1 ratio of compute nodes to I/O nodes, the first compute node has IP address <tt>192.168.1.16</tt>.  Do not blame us for this chaos &#8211; blame IBM&nbsp;:-).
+At least on Argonne machines, with a 64:1 ratio of compute nodes to I/O nodes, compute nodes have addresses <tt>192.168.1.1</tt> to <tt>192.168.1.64</tt> (the last octet of the address is the [[FAQ#Pset rank|pset rank]]).  Somewhat confusingly, the first compute node (compute node <tt>0</tt>) has IP address <tt>192.168.1.64</tt>, so if one submits a one-node job as we did, that is the IP address that needs to be used to log on that sole running compute node.  On a machine with a 16:1 ratio of compute nodes to I/O nodes, the first compute node has IP address <tt>192.168.1.16</tt>.  If you are beginning to see a pattern here, then be advised that with a 64:1 ratio, the IP address of the second compute node is... <tt>192.168.1.59</tt>.  Do not blame us for this chaos &#8211; blame IBM&nbsp;:-).
 The compute nodes are running a <tt>telnet</tt> daemon, and no password is required to log on them:
@@ Line 372: / Line 372: @@
 </pre>
-The IP address of the I/O node on this virtual network is <tt>192.168.1.254</tt>. The network is local to each I/O node, so for larger jobs, there will be multiple distinct virtual networks that cannot communicate with each other, and the IP addresses will duplicate.
+The IP address of the I/O node on this virtual network is <tt>192.168.1.254</tt>. The network is local to each I/O node, so for larger partitions with more than one I/O node, there will be multiple distinct virtual networks that cannot communicate with each other, and the IP addresses will duplicate.
-Here's part of the <tt>ps</tt> output from the compute node:
+Here is part of the <tt>ps</tt> output from a compute node:
 <pre>
@@ Line 395: / Line 395: @@
 '''control''' is a job management daemon responsible for tasks such as the launching of application processes, for the forwarding of stdin/out/err data, and for the management of the virtual network tunneling device from the compute node side.  Do not interfere with this process in any way; this would likely make the node inaccessible.
-'''zoid-fuse''' is a FUSE ([http://fuse.sourceforge.net/ Filesystem in Userspace]) client responsible for making the filesystems from the I/O nodes available to ordinary POSIX-compliant processes running on the compute nodes.  The whole filesystem namespace from the I/O nodes is made available on the compute nodes under <tt>/fuse/</tt>, and symbolic links such as <tt>/home -> /fuse/home</tt> are set up to keep the login and I/O node pathnames valid on the compute nodes.  Please verify that this is correctly set up.  We do not foresee a need to change this setup, but should that prove necessary, the responsbile <tt>fuse-start</tt> and <tt>fuse-stop</tt> scripts can be found under <tt>ramdisk/CN/tree/bin</tt>.
+'''zoid-fuse''' is a FUSE ([http://fuse.sourceforge.net/ Filesystem in Userspace]) client responsible for making the filesystems from the I/O nodes available to ordinary POSIX-compliant processes running on the compute nodes.  The whole filesystem namespace from the I/O nodes is made available on the compute nodes under <tt>/fuse/</tt>, and symbolic links such as <tt>/home -> /fuse/home</tt> are set up to keep the front end and I/O node pathnames valid on the compute nodes.  Please verify that this is correctly set up.  We do not foresee a need to change this setup, but should that prove necessary, the responsbile <tt>fuse-start</tt> and <tt>fuse-stop</tt> scripts can be found under <tt>ramdisk/CN/tree/bin/</tt>.
 ==Shell script job==
@@ Line 417: / Line 417: @@
 (please see the [[FAQ#Pset rank|FAQ]] for the explanation of <tt>/proc/personality.sh</tt> and <tt>BG_RANK_IN_PSET</tt>)
-Please create the script file on the network filesystem, set the executable bit (<tt>chmod 755</tt>) and submit it.  Verify that the script starts correctly and that at least the standard error output is visible immediately.  The scripts print a line of output from each node every ten seconds.  It does so both to the standard output and to the standard error, because, depending on software configuration, the standard output stream could be buffered.  If that is the case, kill the job and verify that the standard output data did appear.
+Create the script file on a network filesystem that is available on the I/O nodes, set the executable bit (<tt>chmod 755</tt>) and submit it.  Verify that the script starts correctly and that at least the standard error output is visible immediately.  The script prints a line of output from each node every ten seconds.  It does so both to the standard output and to the standard error, because, depending on software configuration, the standard output stream could be buffered on the service node.  If that is the case, kill the job and verify that the standard output data did appear.
 ==MPI and OpenMP jobs==
@@ Line 437: / Line 437: @@
 ===Submitting===
-Submit the MPI test like any other job; use one of the below:
+Submit the MPI test like any other job; use one of the below commands:
 <pre>
@@ Line 446: / Line 446: @@
 </pre>
-For the OpenMP test, we pass the number of OpenMP threads to use in the <tt>OMP_NUM_THREADS</tt> variable:
+For the OpenMP test, we pass the number of OpenMP threads to use in the <tt>OMP_NUM_THREADS</tt> environment variable:
 <pre>

@@ Line 6: / Line 6: @@
 ==The /bin/sleep job==
-If you are using Cobalt, submit using either of the commands below:
+If using Cobalt, submit using either of the commands below:
 <pre>
@@ Line 13: / Line 13: @@
 </pre>
-If you are using <tt>mpirun</tt> directly, submit as follows:
+If using <tt>mpirun</tt> directly, submit as follows:
 <pre>

@@ Line 456: / Line 456: @@
 The MPI test benchmarks the performance of various MPI operations.  The OpenMP test is just a parallel "Hello world".
 ----
 [[Installation]] | [[ZeptoOS_Documentation|Top]] | [[MPICH, DCMF, and SPI]]

@@ Line 353: / Line 353: @@
 </pre>
-At least on Argonne machines, with a 64:1 ratio of compute nodes to I/O nodes, compute nodes have addresses <tt>192.168.1.1</tt> to <tt>192.168.1.64</tt> (the last octet of the address is the [[FAQ#Pset rank|pset rank]]).  Somewhat confusingly, the first compute node (compute node <tt>0</tt>) has IP address <tt>192.168.1.64</tt>, so if one submits a one-node job as we did, that is the IP address that needs to be used to log on that sole running compute node.  The IP address of the second compute node is... <tt>192.168.1.59</tt> (do not blame us &#8211; blame IBM&nbsp;:-).
+At least on Argonne machines, with a 64:1 ratio of compute nodes to I/O nodes, compute nodes have addresses <tt>192.168.1.1</tt> to <tt>192.168.1.64</tt> (the last octet of the address is the [[FAQ#Pset rank|pset rank]]).  Somewhat confusingly, the first compute node (compute node <tt>0</tt>) has IP address <tt>192.168.1.64</tt>, so if one submits a one-node job as we did, that is the IP address that needs to be used to log on that sole running compute node.  The IP address of the second compute node is... <tt>192.168.1.59</tt>.  On a machine with a 16:1 ratio of compute nodes to I/O nodes, the first compute node has IP address <tt>192.168.1.16</tt>.  Do not blame us for this chaos &#8211; blame IBM&nbsp;:-).
 The compute nodes are running a <tt>telnet</tt> daemon, and no password is required to log on them:

@@ Line 299: / Line 299: @@
 </pre>
-SSH is supposed to let the partition owner on the I/O node without asking for a password (no other unprivileged user will be allowed on the node), but that might require site-specific customization to work properly (see [[ZOID#The /bin.rd/update_passwd_file.sh file|update_passwd_file.sh]]).  Until this is set up, one might prefer to log on as root (<tt>ssh -l root</tt>), passing the password provided while building the ZeptoOS environment.
+If everything is configured correctly, SSH will only let in root and the partition owner; no other unprivileged user will be allowed on the node.  However, this might require site-specific customization to work properly.  To enable access for the partition owner, one might need to make adjustments to [[ZOID#The /bin.rd/update_passwd_file.sh file|update_passwd_file.sh]].  To enable password-less login for the partition owners without requiring them to set up personal SSH keypairs, we recommend to add the names of the front end nodes to the <tt>shosts.equiv</tt> file, found in <tt>ramdisk/ION/ramdisk-add/etc/ssh.zepto/</tt> (it is empty by default; remember to use the names from the network that interconnects front end and I/O nodes, which might be different from hostnames, e.g., at Argonne we need to add the <tt>-data</tt> suffix to the hostnames).  Until this has all been set up, one might prefer to log on as root (<tt>ssh -l root</tt>), passing the password provided while [[Configuration#Building|building]] the ZeptoOS environment.
 Also, even when the partition owner is correctly set up, there will be a time window while booting the I/O node when the SSH daemon is already running, but a job has not yet been started; during that window, the partition owner cannot log on. If that happens, wait a few seconds and try again.

@@ Line 432: / Line 432: @@
 <pre>
 $ /path/to/install/bin/zmpicc -o mpi-test-linux mpi-test.c
-$ /path/to/install/bin/zmpixlc_r -openmp -o omp-test-linux omp-test.c
+$ /path/to/install/bin/zmpixlc_r -qsmp=omp -o omp-test-linux omp-test.c
 </pre>

@@ Line 9: / Line 9: @@
 <pre>
-cqsub -k <profile-name> -t <time> -n 1 /bin/sleep 3600
+$ cqsub -k <profile-name> -t <time> -n 1 /bin/sleep 3600
-qsub --kernel <profile-name> -t <time> -n 1 /bin/sleep 3600
+$ qsub --kernel <profile-name> -t <time> -n 1 /bin/sleep 3600
 </pre>
@@ Line 16: / Line 16: @@
 <pre>
-mpirun -verbose 1 -partition <partition-name> -np 1 -timeout <time> \
+$ mpirun -verbose 1 -partition <partition-name> -np 1 -timeout <time> \
 -cwd $PWD -exe /bin/sleep 3600
 </pre>
@@ Line 440: / Line 440: @@
 <pre>
-cqsub -k <profile-name> -t <time> -n <number-of-processes> $PWD/mpi-test-linux
+$ cqsub -k <profile-name> -t <time> -n <number-of-processes> $PWD/mpi-test-linux
-qsub --kernel <profile-name> -t <time> -n <number-of-processes>  $PWD/mpi-test-linux
+$ qsub --kernel <profile-name> -t <time> -n <number-of-processes>  $PWD/mpi-test-linux
-mpirun -verbose 1 -partition <partition-name> -np <number-of-processes> -timeout <time> \
+$ mpirun -verbose 1 -partition <partition-name> -np <number-of-processes> -timeout <time> \
 -cwd $PWD -exe $PWD/omp-test-linux
 </pre>
@@ Line 449: / Line 449: @@
 <pre>
-cqsub -k <profile-name> -t <time> -n 1 -e OMP_NUM_THREADS=<num> $PWD/omp-test-linux
+$ cqsub -k <profile-name> -t <time> -n 1 -e OMP_NUM_THREADS=<num> $PWD/omp-test-linux
-qsub --kernel <profile-name> -t <time> -n 1 --env OMP_NUM_THREADS=<num> $PWD/mpi-test-linux
+$ qsub --kernel <profile-name> -t <time> -n 1 --env OMP_NUM_THREADS=<num> $PWD/mpi-test-linux
-mpirun -verbose 1 -partition <partition-name> -np 1 -timeout <time> \
+$ mpirun -verbose 1 -partition <partition-name> -np 1 -timeout <time> \
 -cwd $PWD -env OMP_NUM_THREADS=<num> -exe $PWD/omp-test-linux
 </pre>

@@ Line 1: / Line 1: @@
-[[Installation]] | [[ZeptoOS_Documentation|Top]] | [[The ZeptoOS V2.0 Kernel]]
+[[Installation]] | [[ZeptoOS_Documentation|Top]] | [[MPICH, DCMF, and SPI]]
 ----
@@ Line 458: / Line 458: @@
 ----
-[[Installation]] | [[ZeptoOS_Documentation|Top]] | [[The ZeptoOS V2.0 Kernel]]
+[[Installation]] | [[ZeptoOS_Documentation|Top]] | [[MPICH, DCMF, and SPI]]

@@ Line 410: / Line 410: @@
 while true; do
      echo "Node $BG_RANK_IN_PSET running (stdout)"
-     echo "Node $BG_RANK_IN_PSET running (stderr)" 1>2
+     echo "Node $BG_RANK_IN_PSET running (stderr)" 1>&2
      sleep 10
 done