Direct Access Transport Libraries - v1 and v2
User space libraries/utilities for Direct Access Transport (DAT). DAT is a transport-independent, platform-independent Application Programming Interface that supports RDMA (remote direct memory access) devices. Directory includes packages that include libdat, libdapl, and utilities for both DAT API Spec Version 1.2 and 2.0.
New Feature Summary
1. The new socket CM provider, introduced in 1.2.8 and 2.0.11 packages, assumes homogeneous cluster and will setup the QP's based on local HCA port attributes and exchanges QP information via socket's using the hostname of each node. IPoIB and rdma_cm are NOT required for this provider. QP attributes can be adjusted via the following environment parameters:
DAPL_ACK_TIMER (default=16 5 bits, 4.096us*2^ack_timer. 16 == 268ms)
DAPL_ACK_RETRY (default=7 3 bits, 7 * 268ms = 1.8 seconds)
DAPL_RNR_TIMER (default=12 5 bits, 12 == 64ms, 28 == 163ms, 31 == 491ms)
DAPL_RNR_RETRY (default=7 3 bits, 7 == infinite)
DAPL_IB_MTU (default=1024 limited to active MTU max)
The new socket cm entries in /etc/dat.conf provide a link to the actual HCA device and port. Example v1 and v2 entries for a Mellanox connectx device, port 1:
OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
This new socket cm provider, was successfully tested on the TATA CRL cluster (#8 on Top500) with Intel MPI, achieving a HPLinpack score of 132.8TFlops on 1798 nodes, 14384 cores at ~76.9% of peak. DAPL_ACK_TIMER was increased to 21 for this scale.
2. New v2 definitions for IB unreliable datagram extension (only supported in scm provider, libdaploscm.so.2)
Extended EP dat_service_type, with DAT_IB_SERVICE_TYPE_UD
Add IB extension call dat_ib_post_send_ud().
Add address handle definition for UD calls.
Add IB event definitions to provide remote AH via connect and connect requests
See dtestx (-d) source for example usage model
Latest Packages (see ChangeLog for recent changes):
- md5sum: 098c3efdf812f291449de0253c35d2b9 compat-dapl-1.2.12.tar.gz
- md5sum: 8bcf281049f7ff282202639d4bc523f8 dapl-2.0.15.tar.gz
For 1.2 and 2.0 support on same system, including development, install RPM packages as follow:
- compat-dapl-1.2.12-1
- compat-dapl-devel-1.2.12-1
- dapl-2.0.15-1
- dapl-utils-2.0.15-1
- dapl-devel-2.0.15-1
- dapl-debuginfo-2.0.15-1
Summary of changes for last 2 releases:
- v1,v2 - allow override of /etc/dat.conf via syscondir option
- v1,v2 - fix dapltest transaction test to avoid cleanup before rdma complete
- v1 - add ipath, ehca socket cm provider entries for v1.2, sync with v2.0
- v1,v2 - iWarp, 1 iov on rdma_reads, reduce iov's in dtest, add dat.conf entry
- v1,v2 - add $(DESTDIR) on install/uninstall hooks
- v2 - add new options to dtestx for UD testing
- v2 - IB UD fixes in common code/socket cm provider to allow multiple EP support
For more details see the following:
- documentation/uDAPL_v20.pdf for v2 API specification
- documentation/transition_to_dat20_120406.pdf for details on porting to 2.0
- documentation/DAT_IB_Extensions.pdf for IB extension details.
- documentation/DAT_IW_Extensions.pdf for iWARP extension details.
- documentation/uDAPL_ofed_testing_bkm.pdf for latest uDAPL OFED BKM, updated 8-22-2008
uDAT/uDAPL source git tree is located at git://git.openfabrics.org/~ardavis/scm/dapl.git