Fine-grained task completion

To couple GASPI with task-based runtimes such as OpenMP or OmpSs the tasks need to know when a GASPI data transfer has been successfully finished. This knowledge needs to be available on the receiving as well as on the sending side. Having that knowledge, one is able to decide on buffer validity on the receiving side (allows local read) and buffer reusability on the sending side (allows local overwrite). The main tools to use are GASPI notifications: with the help of notifications, synchronization can be achieved. The sender sends the data together with a notification to the receiver, so the receiver can check on the receipt of the notification with the help of a gaspi_waitsome call, which signals that the data have arrived at the receiver’s side. This allows to check for successful completion on the receiver side.

In the INTERTWinE project we have examined two options to check for the successful completion of the communication on the initiating, i.e. sender side:

1. Acknowledgement mechanism:

Sending out two notifications (sending a notification from the sender to the receiver and on successful completion of the data transfer to send an acknowledgement notification from the receiver back to the sender)

2. Local completion:

Provide GASPI functionality to allow to wait on the local, i.e. sender side completion of the GASPI write to check that the data have been sent from the sender buffer and the buffer can be reused.

The first option is depicted below from the GASPI point of view. The second option would not need a notify from the receiver to the sender.

When combining GASPI with OmpSs, OmpSs will register the tasks (i.e. the sender and the receiver) with a polling service, which is supposed to check and wait for completion of communication.

The two options have different characteristics concerning the latency. Clearly the latency of the notifications is at least two times higher than the latency of Option 2. Both options have been tested in implementations of GASPI and OmpSs to compare the overall performance in both cases.

In terms of the effect on the GASPI standard, Option 1 is the recommended way to use notifications for weak synchronisation in GASPI and does not require any changes in the specification.

Option 2 needs an extension to the GASPI specification. The OmpSs runtime system needs to map communication requests to tasks and vice versa. This could be achieved by tagging communication requests. In such a case a request with a unique tag id is required as additional parameters of the communication. In order to keep backward compatibility of the GASPI standard for all existing applications, we propose to extend the GASPI standard with a communication struct, which stores all the necessary information which can then be used in the communication.  A possible way to introduce this extension has been discussed at the GASPI Forum Meeting in June 2018 in Frankfurt.

Our work on Notified Communication for Shared Windows (GASPI-NOCOS) and Shared Notifications (GASPI-SHAN) has also been presented to the GASPI Forum.