CH3 VC protocol

From Mpich2

This document describes the protocol CH3 uses when closing a VC. The close protocol ensures that both sides of the connection are ready to close and all communication has finished.

Overview

When a VC is released, and its ref count drops to 0, the VC is closed. Similarly in MPID_Finalize(), MPIDI_PG_Close_VCs() is called which goes through all VCs in all process groups and closes them. A VC is closed by calling MPIDI_CH3U_VC_SendClose().

In MPID_Finalize(), MPIDI_CH3U_VC_WaitForClose() is called to wait for all VCs to finish closing. In MPIDI_CH3U_VC_SendClose(), outstanding_close_ops is incremented to keep track of the number of VCs that are in the process of closing. When the close protocol completes for a VC, outstanding_close_ops is decremented. In this way, MPIDI_CH3U_VC_WaitForClose() can simply keep calling progress while outstanding_close_ops is non-zero.

All VCs start in the INACTIVE state at init time. When a message is sent over a VC the VC is connected by the channel, if necessary, and its state is changed to ACTIVE.

State Diagram

Below is the state diagram for the close protocol. There are five states in the protocol: ACTIVE, LOCAL_CLOSE, REMOTE_CLOSE, CLOSE_ACKED and INACTIVE. In the labels for the transitions, the part above the line indicate the event triggering the transition. The part below the line describes the actions taken in the transition. The smaller text following an event or action is the function where the event is handled, or the action is performed if it's not performed by the event handler. The PKT_CLOSE packet has one field called ack.

Graphviz version

This graph is an alternative version in inline graphviz in the wiki. If you want to edit it, simply edit the page and manipulate the dot source accordingly. I (goodell@) created it before I realized that Darius had already written up the one above. It is worth noting that they differ subtly, and I think that it mine is more accurate, at least as of r1265.