ZOID

Other Packages and Utilities | Top | KTAU

---

Introduction

ZOID is an I/O forwarding component of the ZeptoOS project. Any communication between the compute nodes and I/O nodes (job management, file I/O, sockets) is facilitated by ZOID.

ZOID infrastructure consists of:

• Multithreaded zoid daemon on the I/O nodes which performs I/O forwarding for the compute nodes and which also communicates with the service node to perform job management,
• control daemon on the compute nodes which is responsible for job management tasks such as launching application processes, for the forwarding of stdin/out/err data, and for forwarding of IP packets,
• zoid-fuse daemon on the compute nodes which performs file I/O forwarding for POSIX-compliant applications.

User interface

User script

Right before a job starts running, and right after the last process of a job has terminated, ZOID daemon attempts to invoke a user script on I/O nodes. By default, the daemon invokes $HOME/zoid-user-script.sh (this pathname can be changed by an administrator). A single parameter is passed to the script: 1 at the job startup, and 0 at the termination.

Information about the job will be passed to the script in the following environment variables:

ZOID_JOB_EXEC

name of the job executable,

ZOID_JOB_ARGS

job arguments, separated by a :

ZOID_JOB_ENV

job environment variables, separated by a :

ZOID_JOB_ID

BG/P control system job id (Note: this is generally different from the Cobalt job ID; see FAQ for the latter),

ZOID_JOB_GLOBAL_SIZE

number of processes in the job (size of MPI_COMM_WORLD),

ZOID_JOB_LOCAL_SIZE

number of job processes handled by this I/O node,

ZOID_JOB_MODE

0 for SMP, 1 for VN, and 2 for DUAL,

SHELL, PATH, USER, and HOME

will also be set...

Note: the user script is invoked synchronously by the daemon, i.e., the job will not start running until the script terminates. If you need processes to run on I/O nodes while the job is running, start them in the background (&).

File broadcast

A /bin.rd/f2cn command is available on the I/O nodes for a very efficient (hardware-assisted) broadcasting of files to all the compute nodes handled by the given I/O node.

The command takes two arguments:

• absolute pathname to the input file on the I/O node,
• absolute pathname to the output file on the compute nodes.

The input file does not need to be physically on the I/O node; it can be on a network filesystem mounted on the node. The file will be created in the ramdisk of each compute node.

The throughput is in practice limited by how fast the input file can be read; we have seen results in excess of 300 MB/s for files residing in the I/O node ramdisk.

Note: all the compute nodes in the pset must be up and running. Do not use this command on incomplete partitions (e.g., a one-process job on a 64-node partition); you will likely hang the ZOID daemon if you try.

Note2: this feature can safely be used from within a user script, so one can, e.g., pre-stage large binaries, like this:

User script ($HOME/zoid-user-script.sh):

#!/bin/sh

if [ "$1" -eq "1" ]; then
    /bin.rd/f2cn $HOME/large_binary /tmp/large_binary
fi
exit 0

Job script (submitted using Cobalt or mpirun):

#!/bin/sh

chmod 755 /tmp/large_binary
/tmp/large_binary

Performance counters

A /bin.rd/statquery command is available on the I/O nodes for obtaining the performance counters of the I/O daemon.

The command takes a single optional argument:

• the interval between successive queries, in seconds.

If the argument is not provided, the command will terminate after the first query.

Here is the sample output generated:

Timestamp:                      1240439085.688831
Total messages sent:            5767
Total bytes sent:               7619170
Total messages received:        5717
Total bytes received:           72575
IP fwd messages sent:           196
IP fwd bytes sent:              5889
IP fwd messages received:       84
IP fwd bytes received:          6453
Stream messages sent:           65
Stream bytes sent:              520
Stream messages received:       65
Stream bytes received:          1416
Broadcast messages sent:        1
Broadcast bytes sent:           2437906
Internal messages sent:         193
Internal bytes sent:            39524
Internal messages received:     256
Internal bytes received:        1792
Plugin 5 messages sent:         0
Plugin 5 bytes sent:            0
Plugin 5 messages received:     0
Plugin 5 bytes received:        0
Plugin 2 messages sent:         5312
Plugin 2 bytes sent:            5135331
Plugin 2 messages received:     5312
Plugin 2 bytes received:        62914

The meaning of the fields is as follows:

Timestamp

number of seconds and microseconds from the epoch, as returned by gettimeofday(2),

IP fwd

IP packet forwarding between compute nodes and I/O nodes

Stream

stdin/out/err streams,

Broadcast

file broadcasts

Internal

job control messages, etc.

Plugin 5

internal mapping plugin, used by MPI

Plugin 2

unix plugin (POSIX file I/O)

The counters are 64-bit integers, so they will take a while to overflow :-).

Example user script ($HOME/zoid-user-script.sh) that samples the statistics every 60 seconds and writes them to a unique file:

#!/bin/sh

if [ "$1" -eq "1" ]; then
    /bin.rd/statquery 60 >$HOME/zoid_stats.$ZOID_JOB_ID.`hostname` &
fi
exit 0

Administrator interface

The zoid I/O daemon accepts a number of command-line options that can be used to change its behavior. They can be adjusted by editing the ramdisk/ION/ramdisk-add/etc/sysconfig/zoid file and rebuilding the I/O node ramdisk:

ZOID_BUFFER_SIZE (-b)

Specifies the size of the buffers used for messages. Because a separate buffer is needed for a request and a reply, and typically no more than one of these needs to be large, to save memory ZOID supports buffers of two sizes: a small one (4 KB by default) and a large one (4 MB+1 KB by default – the 1 KB is there to accommodate the headers). Use colon (:) to separate the two sizes when customizing this value. If desired, support for two separate buffer sizes can be disabled by providing only one value to this option.

ZOID_ACK_THRESHOLD (-a)

Specifies a size threshold for the rendezvous protocol for messages coming from the compute nodes, in the units of tree network packets (240 bytes each). An eager protocol is used for messages below the threshold. Messages above the threshold use flow control in the form of a rendezvous protocol with message acknowledgements; basically, the daemon will only receive one large message at a time, which improves the predictability and an overall throughput. The daemon default for this option is to not use acknowledgements, but the config file defaults to a value of 8, which is the size of the hardware FIFO buffer of the tree network device. Set this option to 0 (or comment it out altogether) to disable message acknowledgements.

ZOID_MODULES (-m)

Specifies a :-separated list of ZOID plugins to load. This defaults to "unix_impl.so:unix_preload.so:mapping_impl.so:mapping_preload.so" in the config file; do not remove any of these or basic system services will stop working. The unix plugin provide POSIX file I/O support, while mapping is used by our MPI implementation to map between MPI ranks and Blue Gene X/Y/Z/T coordinates. Custom plugins can be created and added here; see Programmer interface for details.

ZOID_ENABLE_NAT (-n)

Enables network address translatation (NAT) for IP packets coming from the compute nodes, allowing compute nodes to communicate with the outside world. This support is disabled by default because we have found that it has a detrimental effect to the overall performance of the TCP/IP stack on the I/O nodes, slowing down network filesystems. This feature can also be enabled on per-job basis by setting the ZOID_ENABLE_NAT environment variable when submitting a job (see the FAQ).

ZOID_USER_SCRIPT (-u)

Specifies the pathname to the user script; it defaults to "/bin.rd/zoid-user-script.sh". This script can be found in ramdisk/ION/ramdisk-add/bin/zoid-user-script.sh; it sets a few environment variables and then invokes user's custom $HOME/zoid-user-script.sh. Hence, if you want to adjust the behavior of this option, you can either change this option or the script in the ramdisk.

Programmer interface

ZOID is a flexible, extensible, high-performance function call forwarding (RPC) infrastructure. Built-in features and the standard plug-ins provide familiar POSIX file I/O and BSD socket interfaces, but, because of the number of software layers involved, they introduce a significant overhead. For applications requiring maximum bandwidth between the compute and I/O nodes, ZOID provides an option of a customized function call forwarding with minimal overheads. This section provides an overview of how to create such a custom plug-in.

Overview

All that ZOID provides is a function call forwarding support, and a limited one at that. Any logic (caching, prefetching, etc.) needs to be custom-built on top of it.

Follow existing plug-ins, found in packages/zoid/src/, as examples. The unix plug-in is generally the most up to date, but other plug-ins such as mapping, zoidfs, and barrier should also be fine.

A plug-in consists of automatically generated client-side and server-side stubs (which perform the marshalling and demarshalling of function call parameters and results, the forwarding of the function call, etc.), and of a hand-written server-side implementation which provides the implementation code for the forwarded function calls. One might also decide to provide hand-written client-side wrappers to hide some details of the ZOID API (such as the error handling) or to adhere to a particular existing API, as is the case with the unix plug-in (the wrappers used by the FUSE client are available in packages/zoid/src/unix/stubs/; another version is in the GNU libc sources, in packages/glibc/src/zoid/sysdeps/unix/sysv/linux/powerpc/powerpc32/).

The scanner.pl script, found in packages/zoid/src/, creates the automatically-generated client and server stubs based on a hand-written input header file described below. Again, please follow the examples from the existing plug-ins, such as unix or mapping. The Makefile in those plug-ins is written in a generic fashion and should only require a change to the PREFIX line to be usable with another plug-in. Use that Makefile to invoke the scanner.pl script and to compile the generated source files.

Input header file

The input header file must be a valid C header file with additional hints in the comments. The file is read by the scanner.pl script.

The parser in the script is rather limited and does not handle many C constructs. It is thus essential that the header file be as simple as possible. In particular, function prototypes should be specified at the end of the file, not intermixed with any other specifications such as data type definitions.

Ordinary comments are best placed on separate lines.

Note: the parse is case sensitive.

Start line

Any complex declarations that the scanner cannot parse should be placed at the top of the file, because the parser ignores everything until it encounters the following magic start line:

/* START-ZOID-SCANNER ID=<n> INIT=<s1> FINI=<s2> PROC=<s3> */

ID=<n>

Each plug-in needs a unique, 16-bit identifier, passed in <n>. The following identifiers are already in use: 0 (internal), 1 (zoidfs plug-in), 2 (unix), 3 (lofar), 4 (test), 5 (mapping), and 10 (ftb).

INIT=<s1>

<s1> provides a name of an initialization function which will be invoked before a job starts running; see Start-line functions for more information. If a plug-in does not need this feature, please specify INIT=NULL.

FINI=<s2>

<s2> provides a name of a termination function which will be invoked after all job processes have exited; see Start-line functions for more information. If a plug-in does not need this feature, please specify FINI=NULL.

PROC=<s3>

<s3> provides a name of a callback function which will be invoked on a startup and termination of every application and ZOID-enabled process; see Start-line functions for more information. If a plug-in does not need this feature, please specify PROC=NULL.

Argument hints

Hints are generally needed to correctly encode and decode function arguments. They are placed after each argument, before a separating comma (or a closing bracket), and are embedded inside dedicated C comments. Multiple hints per argument are usually provided; these are separated by a colon (:). The following hints are currently defined:

in, out, inout

Specifies whether the argument is an input argument, an output argument, or both. in is the default.

obj, str, ptr, arr, arr2d

Specifies the type of the argument, respectively a plain object (say, an int, or a structure passed by value), a '\0'-terminated character string, a pointer to an object, an array of objects, or a two-dimensional array (type**, not type[][]). obj is the default.

size

Required for array arguments (arr and arr2d). Indicates the index of another argument in the same function, which is used to pass the array size. Absolute numbers are accepted (1 to number of arguments) or relative ones (+1 for the next argument, -1 for the previous argument, etc).
For arr arguments, the size argument must be of a numerical type, or a pointer to such a type. For arr2d arguments, the size argument must itself be an array (an arr argument) of numerical elements, specifying the sizes along the less significant dimension of the array (the size of the more significant dimension is the size of the arr array itself).
Please note that the unit of size for the numerical types is the size of the base array type (thus, sizeof(int) for an array of ints), not byte (if you want it to be byte, just make the array argument have type char* or void* (a GCC extension)).

nullok

An option for arguments passed by pointer (basically, all but obj). If provided, it indicates that the argument is allowed to be NULL. This is not the default because supporting NULL pointers results in an additional computational and protocol overhead. If a NULL pointer is passed to an argument without the nullok flag, the client will crash.

zerocopy

An option for array arguments. Enables a more efficient marshalling/demarshalling protocol for the array, which does not use extra memory copies. Can be used for no more than one in argument and no more than one out argument. Zerocopy performance discusses performance considerations of using this option.

userbuf

An option for zerocopy; only supported for arr arguments. Enables a special form of zero-copy support, discussed in Zerocopy with custom output buffer and Zerocopy with custom input buffer.

An example function prototype:

int zoidfs_readlink(const zoidfs_handle_t * handle /* in:ptr */,
                    char * buffer /* out:arr:size=+1 */,
                    size_t buffer_length /* in:obj */);

Limitations

As indicated earlier, the scanner is limited, so keep the prototypes simple.

Return type of a forwarded function must be a scalar or void.

Structures with pointer fields inside of them cannot be forwarded.

Generated files

For every function prototype found, the scanner generates two output files: one for a client calling the function and one for the server, where the function is in fact executed. Code in the generated files performs marshalling and demarshalling of function arguments and results.

Two more files per plug-in are generated: header_defs.h and header_dispatch.c.

None of the generated files should be edited.

Server-side API

Server-side stubs and the server-side implementation need to be passed as modules when invoking the ZOID I/O daemon, as described earlier.

The hand-written server-side implementation code should include the zoid_api.h header file (available from packages/zoid/prebuilt/) and the plug-in input header file.

All the functions listed in the header file need to be defined the the server-side implementation code. The code needs to be compiled as a shared library; use the implementation/ subdirectory of the unix plug-in as an example. Please note that since ZOID is multi-threaded, multiple functions can be invoked at the same time, so you must ensure that your implementation is multi-thread-safe.

Start-line functions

The following start-line function can be defined:

void INIT(int pset_mpi_proc_count, int argc, int envc, const char* argenv);

The INIT function is invoked during initialization, right before a job starts running. Arguments:

pset_mpi_proc_count

The number of job processes that will be handled by this I/O node. Note that I/O nodes also handle additional ZOID-enabled processes, such as the FUSE clients, which are not included in this number.

argc

The number of command-line arguments plus one.

envc

The number of environment variables.

argenv

An array of '\0'-terminated strings, one after another. The first string is the name of the job executable, followed by argc-1 command-line arguments, followed by envc environment variables.

void FINI(void);

The FINI function is invoked after the last process of the job has terminated.

void PROC(int added, int pset_pid);

The PROC function is invoked on the startup and termination of every application and ZOID-enabled process on the compute node. Arguments:

added

1 if the process was started, 0 if it was terminated.

pset_pid

A process identifier (as returned by __zoid_calling_process_id).

Implementation functions

The hand-written server-side implementation functions can themselves call back a few ZOID functions, available by including the zoid_api.h header file:

int __zoid_calling_process_id(void);

This function returns a unique identifier of the compute node process that invoked the function. The identifier is not an MPI rank, because some processes, such as the FUSE clients, are not part of the application and hence do not have a rank. The identifiers are only unique within one I/O node, and they can be reused if one process terminated before another one started.

void __zoid_register_userbuf(void* userbuf,
                             void (*callback)(void* userbuf, void* priv),
                             void* priv);

This function will be discussed in Zerocopy with custom output buffer.

int __zoid_send_output(int pid, int fd, const char* buffer, int len);

This function writes an arbitrary string to the job's standard output or error. Arguments:

pid

Process identifier as returned by __zoid_calling_process_id. The process in question must have an MPI rank, meaning that it must be either an application process or a process launched from an application process.

fd

1 for standard output, 2 for standard error.

buffer, len

The string and its length. '\0' should not be included in len and buffer does not need to be '\0' terminated.

The function returns 0 if successful, and -1 if not (such as when the process identified by pid does not have an MPI rank).

Client-side API

A compute node application needs to be linked with the client-side stubs and with a common support library libzoid_cn.a (a prebuilt version of the latter is in packages/zoid/prebuilt; sources are in packages/zoid/src/cnl/client). Several functions are available to applications by including the zoid_api.h header file:

Initialization

int __zoid_init(void);

This function must be invoked before any ZOID or ZOID-forwarded functions can be invoked. It returns 0 if successful, 1 otherwise. There is no corresponding termination function.

int __zoid_job_size(void);
int __zoid_my_rank(void);

These functions return, respectively, the number of processes in the job (the size of MPI_COMM_WORLD), and the MPI rank of the current process. Either will return -1 if the current process does not have an MPI rank, i.e., if it is not an application process and was not launched from an application process (say, if it was launched from an interactive shell).

Error conditions

int __zoid_error(void);

This function should be invoked on the client side after every forwarded function call returns, to determine if any errors occured within the forwarding layer. A return value of 0 indicates a success; otherwise, one of the following error values will be returned:

ENOSYS

Invalid command sent from the client. Typically indicates that the corresponding I/O-node-side modules have not been loaded.

ENOMEM

Out of memory condition.

E2BIG

Message exceeded the internal size limit.

int __zoid_excessive_size(void);

If __zoid_error returned E2BIG, calling this function will provide an indication by how many bytes the input or output was too large.

ZOID has a limit on the message size, around 4 MB by default. The limit is enforced on both input and output. The limit only applies to buffers "owned" by ZOID on the daemon side; it does not apply to custom input or output buffers.

If the limit is hit, the operation needs to be split into smaller ones. Information returned by __zoid_excessive_size makes it easy to adjust the buffer and resubmit.

Note: While the input-side (argument) overflow is flagged immediately on the client side, and is thus fairly cheap to hit, the output-side (result) overflow is flagged on the I/O node, after the request has been sent there (but before the implementation function is invoked). It is thus advised to cache at least the size limit for the output side for the next invocation, to avoid a future communication overhead. The size limit is function-specific, since it depends on sizes of other arguments and results.

Here is an example of how the client-side convenience wrapper for a call such as POSIX read could be implemented:

ssize_t read(int fd, void *buf, size_t nbytes)
{
    static ssize_t max_read_nbytes = -1;
    ssize_t bytes_read;

    bytes_read = 0;
    do
    {
        ssize_t toread, justread;
        int error;

        toread = nbytes - bytes_read;

        if (max_read_nbytes != -1 && toread > max_read_nbytes)
            toread = max_read_nbytes;

        /* unix_read is the forwarded function call.  */
        justread = unix_read(fd, buf + bytes_read, toread);

        if ((error = __zoid_error()))
        {
            if (error != E2BIG)
            {
                /* For a generic ZOID error, just bail out.  */
                __set_errno(error);
                return -1;
            }

            /* We tried to send a too large read request.  Adjust.  */
            max_read_nbytes = toread - __zoid_excessive_size();
        }
        else
        {
            if (justread < 0)
            {
                /* For a generic read() error, just bail out.
                   In case of an I/O error, unix_read returns -errno.  */
                __set_errno(-justread);
                return -1;
            }

            bytes_read += justread;

            if (justread != toread)
                /* unix_read as such succeeded, but it read fewer bytes than
                   expected.  We terminate prematurely then.  */
                break;
        }
    } while (bytes_read < nbytes);

    return bytes_read;
}

Additional considerations

Returning variable amounts of data in arrays

Array outputs introduce extra restrictions. All the other results must have sizes that can be calculated before a function is invoked. For arr2d outputs, that also applies to each array line: their sizes cannot change inside the invoked function, if there are any more non-zero-size array lines to follow. E.g, if the size array of an arr2d is "inout", and initially holds {10, 10, 10}, then after the function returns, it is allowed to hold {0, 0, 0}, or {5, 0, 0}, or {10, 1, 0}, or {10, 10, 10}, but *not* {10, 9, 10} (because the modified array line size is followed by another (non-zero-size) line). Such restrictions apply to 1-D arrays only if they are followed by another output array. The reason behind these restrictions is that the placement of arrays in result buffer is fixed before a function is invoked, to avoid additional buffering and copies.

Zerocopy performance

Because of the additial protocol overheads it introduces, it should be used only for potentially large memory buffers.

Zerocopy with custom input buffer

---

Other Packages and Utilities | Top | KTAU