Notified Communication for Shared Windows (GASPI-NOCOS)

The use of shared memory offers the opportunity to move away from a process centric communication model towards a node centric communication model. While MPI-3 readily supports this model with MPI 2-sided and 1-sided communication, we here aim support notified, one-sided communication in GASPI. There is an ongoing effort to implement collectives such as allreduce, allgather(v) and alltoall(v) on top of this model. First results are promising, as notified communication so far has not been used for the level of collectives. We refer to this model as “NOtified COmmunication for Shared Windows. (GASPI-NOCOS).

GASPI supports single-sided and weakly synchronized communication. The communication partners use notifications, which allow checking whether a communication has finished successfully. Notifications are stored in a notification area connected to the segment (a designated continuous array of memory inside the memory of a node, which is defined in advance). Normally memory is allocated by process but is readable and writable for all processes participating in the corresponding procedure call. To allow shared notifications the segment is complemented with a shared notification space, see below. Every local process/rank with access to the common shared memory then becomes able to read/write the common notification space. This implies that every local process can see all incoming messages for all other local ranks with access to the shared application window.

The interface is now able to aggregate data for remote nodes and to perform one single write to the other node (for all local processes on that node) and notify all remote local processes in one step. As all remote processes can detect and access this common buffer, each remote process/rank can retrieve the required partial data for its ghost cell exchange. Therefore, this generic interface will facilitate the interoperability of MPI and GASPI significantly.

To use shared notifications user allocated memory for segments is required. The allocated memory can be a MPI shared window. The binding to a GASPI segment happens with the gaspi_segment_use command with the policy GASPI_NODE_LOCAL.

The shared notification proposal does not necessitate any changes in the GASPI standard except for the introduction of a new memory allocation policy for GASPI segments, which is already allowed in the GASPI specification.

Our work on Fine-grained task completion and Shared Notifications (GASPI-SHAN) has also been presented to the GASPI Forum.