GASPI-SHAN: GASPI Shared Notifications

This Resource Pack is an extension to the Resource Pack on GASPI plus MPI for communication/ performance critical parts of codes, aiming to expose their data types in shared memory.

This page has the following sections:

Motivation and Strategy

In the INTERTWinE Deliverable D5.3, IPIC3D and Ludwig have pointed out the path to interoperability of MPI-only code to GASPI is cumbersome. This is due to the fact that in GASPI concurrency within a compute node is typically handled by threads whereas many MPI application use a flat MPI model. To mitigate the effect, we propose the GASPI-SHAN library which handles the data in shared memory (within a node). The GASPI-SHAN library can then be coupled to the newly introduced shared notification policyfor internode communication provided in the GPI implementation.

GASPI-SHAN directly exposes application data and data types in shared memory. While the concept of a shared memory implementation to some extent has been available in MPI since MPI-3.0, the latter concept (exposing data types) does not yet exist in MPI.

We have extended the notified communication of GASPI to shared memory in the form of SHAred memory Notifications (or in short: SHAN). Shared memory notifications allow for a fine grain mechanism, which is suitable for mutual process dependencies in shared memory. This programming model opens a migration path for existing flat MPI legacy applications towards an asynchronous data flow model in which explicit communication handshake mechanisms are replaced with mutual data dependencies and corresponding notified communication (either intra-node (SHAN) or inter-node (GASPI)). An extended presentation can be found in the INTERTWinE News Announcement

The SHAN library is available under: 

Best Practice Guide

Best Practice Guide for the migration of MPI legacy code towards the GASPI-SHAN data-flow model serves as a guide for application developers who are considering an improved scalability for their applications via notified intra- and inter-node communication. To support this, the INTERTWinE project has developed a GASPI shared memory extension (GASPI-SHAred Notifications, GASPI-SHAN), which extends the notified communication of GASPI into shared memory windows.

GASPI-SHAN Benchmarks

The INTERTWinE team has ported several benchmarks to illustrate applicability and benefits of the GASPI-SHAN approach, along with developers’ commentary. These benchmarks are Heat, MiniGhost, CFD proxy benchmarks. We see good performance of the GASPI-SHAN implementation with a clear trend: the more cores we have per node, the higher the relative performance gain of GASPI-SHAN vs MPI. We also note that the GASPI-SHAN implementation (due to its relatively small per-node overhead) is able to make more efficient use of hyperthreading than MPI.

In order to validate this new strategy with the new programming paradigm of shared notifications in GASPI, we implemented an equivalent to the MPI Allreduce. Two algorithms were implemented: pipelined rings and a dissemination algorithm. As another test case, we choose Allgather(V) that is an appealing collective due to the variable size of the incoming data from every process.

Software to support this release pack can be downloaded from https://github.com/EPCCed/GASPI_SHAN

Resource Pack

The INTERTWinE GASPI-SHAN Resource Pack contains the following list of benchmarks and collective operations, shipped together with guides and codes:

  1. INTERTWinE Best Practice Guide for the migration of MPI legacy code towards the GASPI-SHAN data-flow model
  2. INTERTWinE developers' commentary on several mini-applications, kernels, and collectives:
    1. The Heat simulation uses an iterative Gauss-Seidel method to solve the heat equation, which is a parabolic partial differential equation that describes the distribution of heat (or variation in temperature) in a given region over time. Guide, Source Code
    2. MiniGhost is a finite Difference code which implements a difference stencil across a homogeneous three-dimensional domain. Guide, Source Code
    3. The CFD Proxy benchmark is a proto-app for a multigrid CFD solver and has been developed within the scope of the EXA2CT (EU FP7) project. The proxy serves as a means to replicate the current programming model of TAU and to also implement, test and evaluate a large variety of hybrid programming models. Guide, Source Code
    4. AllReduce collective operation. Code is available upon request. [Guide]
    5. AllGather(V) collective operation. Code is available upon request. [Guide]

For more details, please consult our deliverables:

  1. D5.4 Final report on application/ kernel plans, evaluations and benchmark suite releases
  2. D5.3 Performance evaluation report
  3. D5.2 Interim report on application/kernel plans, evaluations and benchmark suite releases
Last updated: 27 Nov 2018 at 9:34