Direct Access Transport Libraries - v1 (02/23/10) and v2 (02/23/2010)

User space libraries/utilities for Direct Access Transport (DAT). DAT is a transport-independent, platform-independent Application Programming Interface that supports RDMA (remote direct memory access) devices. Directory includes packages that include libdat, libdapl, and utilities for both DAT API Spec Version 1.2 and 2.0.

New Feature Summary (introduced in OFED 1.4, 12/10/2008)

1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages, assumes homogeneous cluster and will setup the QP's based on local HCA port attributes and exchanges QP information via socket's using the hostname of each node. IPoIB and rdma_cm are NOT required for this provider. QP attributes can be adjusted via the following environment parameters:

DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 == 268ms)

DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds)

DAPL_RNR_TIMER (default=12 5 bits, 12 == 64ms, 28 == 163ms, 31 == 491ms)

DAPL_RNR_RETRY (default=7 3 bits, 7 == infinite)

DAPL_IB_MTU (default=1024 limited to active MTU max)

The new socket cm entries in /etc/dat.conf provide a link to the actual HCA device and port. Example v1 and v2 entries for a Mellanox connectx device, port 1:

OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""

ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""

This new socket cm provider, was successfully tested on the TATA CRL cluster (#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on 1798 nodes, 14384 cores at ~76.9% of peak. DAPL_ACK_TIMER was increased to 21 for this scale.

2. New v2 definitions for IB unreliable datagram extension (only supported in scm provider, libdaploscm.so.2)

Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD

Add IB extension call dat_ib_post_send_ud().

Add address handle definition for UD calls.

Add IB event definitions to provide remote AH via connect and connect requests

See dtestx (-d) source for example usage model

Latest Packages (see ChangeLog for recent changes):

md5sum: ee6540f0ccd777a45b8b7b17c3b49b36 compat-dapl-1.2.16.tar.gz
md5sum: 999734fc1f2e27b5fc98dd6fd486daf9 dapl-2.0.27.tar.gz

For 1.2 and 2.0 support on same system, including development, install RPM packages as follow:

dapl-2.0.27-1
dapl-utils-2.0.27-1
dapl-devel-2.0.27-1
dapl-debuginfo-2.0.27-1
compat-dapl-1.2.16-1
compat-dapl-devel-1.2.16-1

Summary of v2.0 and v1.2 changes:

Release 2.0.27 fixes (OFED 1.5.1):
- windows: add scm makefile
- Windows does not require rdma_cma_abi.h, move the include from common code
- Windows patch to fix IB_INVALID_HANDLE name collision
- scm: dat_ep_connect fails on 32bit servers
- undefined symbol: dapls_print_cm_list
- Cleanup CM object lock before freeing CM object memory
- destroy verbs completion channels created via ia_open or ep_create.
- Update Copyright file and include the 3 license files in distribution
- When copying private_data out of rdma_cm events, use the
- dapl/cma: fix referencing freed address
- dapl: move close device after async thread is done
Release 1.2.16 fixes (OFED 1.5.1):
- Update Copyright file and include the 3 license files in distribution
- cma: max sge incorrectly decremented during ibv_device_query
For more details see the following:
- documentation/uDAPL_release_notes.txt for latest OFED release notes
- documentation/uDAPL_v20.pdf for v2 API specification
- documentation/transition_to_dat20_120406.pdf for details on porting to 2.0
- documentation/DAT_IB_Extensions.pdf for IB extension details.
- documentation/DAT_IW_Extensions.pdf for iWARP extension details.
- documentation/uDAPL_ofed_testing_bkm.pdf for latest uDAPL OFED BKM, updated 8-22-2008
uDAT/uDAPL source git tree is located at git://git.openfabrics.org/~ardavis/scm/dapl.git