Other Packages
PVFS
IP over torus
This is currently a preview feature. It implements IP packet forwarding on top of MPI, over the torus network. Torus is a point-to-point network that interconnects all the compute nodes in a partition. Every compute node gets a unique IP address, of the form:
10.128.0.0 | <rank>
where <rank> is the MPI rank. Thus, for a 64-node partition, the IP addresses will range between 10.128.0.0 and 10.128.0.63, and for a 1024-node partition, they will range between 10.128.0.0 and 10.128.3.255.
To try this feature out, submit as a compute job the cn-ipfwd.sh script, which should have been installed in /path/to/install/bin. The script initializes the IP forwarding and then goes to sleep; feel free to adjust it to your needs. Log into a compute node, and run ifconfig; there should be a new virtual tun1 network (in addition to the usual tun0, used for IP forwarding between compute nodes and I/O nodes):
~ # ifconfig tun1 ~ # ping 10.128.0.1 ~ #
This feature can be used to implement an arbitrary IP-based network protocol between the compute nodes. We have even experimented running a TCP/IP-based MPICH on top of it (which, while obviously not as fast as the native Blue Gene one, has the advantage of being able to, e.g., run multiple MPI jobs at a time on a single partition).
One major disadvantage of this feature is that the current implementation is computationally intensive; it permanently occupies one core on each node.
ZOID glibc
This is another preview feature. It provides a modified version of GNU libc for the compute nodes, which features much better file I/O throughput rates to the I/O nodes and remote file systems than the default one. It does so by communicating with the ZOID daemon directly, instead of going through the Linux kernel and the FUSE client (which, while convenient, is slow).
The modified glibc is meant for compiled application processes, not for shell scripts and such. It is currently only available in a static (.a) version. It is installed with the rest of the ZeptoOS, in /path/to/install/lib/zoid/. To link with it, simply add -L/path/to/install/lib/zoid to the final linking stage. Use the following command to verify that the modified version of glibc has been used:
$ nm binary | grep __zoid_init
