FAQ - Revision history

Iskra: /* How to obtain a Cobalt job ID */

2010-05-06T22:49:12Z

How to obtain a Cobalt job ID

Iskra: /* How to open a socket from a CN to the outside world */

2009-10-19T22:24:14Z

How to open a socket from a CN to the outside world

Iskra at 22:26, 8 May 2009

2009-05-08T22:26:26Z

Iskra at 21:21, 7 May 2009

2009-05-07T21:21:52Z

Iskra: /* Why large MPI processes do not work */

2009-05-07T20:51:30Z

Why large MPI processes do not work

Iskra: /* Why large MPI processes do not work */

2009-05-07T20:48:33Z

Why large MPI processes do not work

Iskra: /* Why large MPI programs do not work */

2009-05-07T20:43:36Z

Why large MPI programs do not work

Iskra at 20:32, 7 May 2009

2009-05-07T20:32:15Z

Iskra at 22:04, 30 April 2009

2009-04-30T22:04:29Z

Iskra at 23:17, 28 April 2009

2009-04-28T23:17:38Z

@@ Line 115: / Line 115: @@
 <pre>
-COBALT_JOBID=`echo $ZOID_JOB_ENV | sed 's/^.*COBALT_JOBID=\([^:]*\)/\1/'`
+COBALT_JOBID=`echo $ZOID_JOB_ENV | sed 's/^.*COBALT_JOBID=\([^:]*\).*$/\1/'`
 </pre>

@@ Line 106: / Line 106: @@
 The most transparent solution to this problem is to perform network address translation (NAT) on the I/O nodes using the Linux kernel netfilter infrastructure.  We used to enable this by default, but experiments have shown it to have a detrimental effect on the overall performance of the TCP/IP stack on the I/O nodes, slowing down access to the network filesystems.
-To enable the translation, pass <tt>ZOID_NAT_ENABLE</tt> environment variable when submitting a job.  An administrator can also enable this option permanently in the [[ZOID#opt_enable_nat|config file]].
+To enable the translation, pass <tt>ZOID_ENABLE_NAT</tt> environment variable when submitting a job.  An administrator can also enable this option permanently in the [[ZOID#opt_enable_nat|config file]].
 ==How to obtain a Cobalt job ID==

@@ Line 71: / Line 71: @@
 ===MPI rank===
-MPI rank should not be confused with a torus rank, even though by default the two are the same.  MPI rank is a property of a process, ''not'' node.  If one submits a job in the <tt>VN</tt> or <tt>DUAL</tt> mode, there will be multiple MPI tasks per node, obviously each with a different MPI rank.  Also, using the <tt>BG_MAPPING</tt> environment variable changes the mapping between the torus coordinates and MPI ranks.
+An MPI rank should not be confused with the torus rank, even though by default the two are the same.  MPI rank is a property of a process, ''not'' node.  If one submits a job in the <tt>VN</tt> or <tt>DUAL</tt> mode, there will be multiple MPI tasks per node, obviously each with a different MPI rank.  Also, using the <tt>BG_MAPPING</tt> environment variable changes the mapping between the torus coordinates and MPI ranks.
 While obtaining MPI rank from an MPI application is trivial, how to obtain it from a shell script?
@@ Line 90: / Line 90: @@
 </pre>
-(compile with <tt>-I</tt>''path_to_ZeptoOS''<tt>/packages/zoid/prebuilt -L</tt>''path_to_ZeptoOS''<tt>/packages/zoid/prebuilt -lzoid_cn</tt>)
+(compile with <tt>-I</tt>''path_to_ZeptoOS_source''<tt>/packages/zoid/prebuilt -L</tt>''path_to_ZeptoOS_source''<tt>/packages/zoid/prebuilt -lzoid_cn</tt>)
 A slight disadvantage of this approach is that <tt>__zoid_init</tt> registers the process with the ZOID daemon on the I/O node, which is an overhead we do not need.  Another solution, without using any binaries, is as follows:

@@ Line 123: / Line 123: @@
 See the [[Kernel#Kernel (command line) parameters|Kernel]] section to learn how to increase the limit; the parameter to use is <tt>flatmemsizeMB</tt>.  We suggest creating multiple profiles with different big memory sizes to accommodate different uses of ZeptoOS.
 ----
 [[ZeptoOS_Documentation|Top]]

@@ Line 120: / Line 120: @@
 ==Why large MPI processes do not work==
-A common reason might be that they do not have enough memory to run.  MPI processes run within the Big Memory region, which by default is limited to just 256&nbsp;MB so as not to deplete the ordinary Linux paged memory pool too much (main memory is allocated to the Big Memory region at boot time and it cannot be used for paged memory, even if it were unused).
+A common reason might be that they do not have enough memory to run.  MPI processes run within the big memory region, which by default is limited to just 256&nbsp;MB so as not to deplete the ordinary Linux paged memory pool too much (main memory is allocated to the big memory region at boot time and it cannot be reclaimed by the kernel, even if it were unused).
-See the [[Kernel#Kernel (command line) parameters|Kernel]] section to learn how to increase the limit; the parameter to use is <tt>flatmemsizeMB</tt>.
+See the [[Kernel#Kernel (command line) parameters|Kernel]] section to learn how to increase the limit; the parameter to use is <tt>flatmemsizeMB</tt>.  We suggest creating multiple profiles with different big memory sizes to accommodate different uses of ZeptoOS.
 ----
 [[ZeptoOS_Documentation|Top]]

@@ Line 118: / Line 118: @@
 </pre>
-==Why large MPI programs do not work==
+==Why large MPI processes do not work==
-A common reason might be that they do not have enough memory to run.  MPI programs run within the Big Memory region, which is limited to 256&nbsp;MB by default.  See the [[Kernel#Kernel (command line) parameters|Kernel]] section to learn how to change that; the parameter to use is <tt>flatmemsizeMB</tt>.
+A common reason might be that they do not have enough memory to run.  MPI processes run within the Big Memory region, which is limited to 256&nbsp;MB by default.  See the [[Kernel#Kernel (command line) parameters|Kernel]] section to learn how to change that; the parameter to use is <tt>flatmemsizeMB</tt>.
 ----
 [[ZeptoOS_Documentation|Top]]

@@ Line 1: / Line 1: @@
-[[KTAU]] | [[ZeptoOS_Documentation|Top]]
+[[ZeptoOS_Documentation|Top]]
 ----
@@ Line 119: / Line 119: @@
 ----
-[[KTAU]] | [[ZeptoOS_Documentation|Top]]
+[[ZeptoOS_Documentation|Top]]

@@ Line 8: / Line 8: @@
 ===Pset rank===
-A pset rank is a number identifying a compute node within each ''pset'' (an I/O node and the compute nodes that communicate with it).  Pset rank is used as the last octet in the IP address on the tree network connecting the compute nodes and the I/O nodes (''x'' in <tt>192.168.1.</tt>''x'').
+A pset rank is a number identifying a compute node within each ''pset'' (an I/O node and the compute nodes that communicate with it).  Note that on partitions larger than one pset, the pset ranks will not be unique.  Also, pset ranks do ''not'' start from <tt>0</tt>; they start from <tt>1</tt> for some mysterious reason (do not blame us &#8211; blame IBM&nbsp;:-).
 The pset rank is available on the compute nodes from <tt>/proc/personality.sh</tt>, in the <tt>BG_RANK_IN_PSET</tt> variable:
@@ Line 54: / Line 56: @@
 (compile the above with <tt>-I/bgsys/drivers/ppcfloor/arch/include</tt>)
-==How to find the MPI rank from a shell script==
+===Torus rank===
 ==How to open a socket from a CN to the outside world==