Figure~\ref{fig:architecture} highlights the general architecture of the
two main OFI components, the libfabric library
and an OFI provider, as they are situated between OFI enabled applications
and a hypothetical NIC that supports
process direct I/O.

The libfabric library defines the interfaces used by applications, and
provides some generic services.
However, the bulk of the OFI implementation resides in the providers.
Providers plug into libfabric and supply access to fabric hardware
and services.
Providers are often associated with a specific hardware device or NIC.
Because of the structure of libfabric, applications access the
provider implementation directly for most operations in order to
ensure the lowest possible software latencies.

As captured in
Figure~\ref{fig:architecture},
libfabric can be grouped into four main services.

\subsection{Control Services}
These are used by applications to discover information about the types of
communication services available in the system.
For example, discovery will indicate what fabrics are reachable
from the local node, and what sort of communication each fabric provides.

The discovery services are defined such that an application can request
specific features, referred to as capabilities,
from the underlying provider, for example, the desired communication model.
In responding, a provider can indicate what additional capabilities
an application may use without negatively impacting performance
or scalability.
In order to ensure the tightest semantic fit between the application software
and the underlying fabric hardware,
a provider can use the discovery interface to inform applications
on how it may best be used.
It does this by setting mode bits which specify limitations on how a provider
should be used in order to provide the best performance.
Mode bits are restrictions that often have a lower impact on performance
if implemented by higher level software than at the provider level.

The result of the discovery process is that a provider uses the application's
request to select a software path that is best suited for that
application's needs.

\subsection{Communication Services}
These services are used to set up communication between nodes.
They include calls to establish connections (connection management)
as well as the functionality used to address connectionless endpoints
(address vectors).

Connection interfaces are integrated directly into libfabric,
which allows hiding fabric and hardware specific details used to connect and
configure communication endpoints.
Although libfabric does not define a connection protocol,
it supports a 3-way handshake that is commonly used. Libfabric
also allows for applications to exchange application specific data as part
of their connection setup.

Address vectors are designed around minimizing the amount of memory
needed to store addressing data for potentially millions of remote peers.
A remote peer can be referenced by simply assigning it an index into
an address vector, which maps well to applications such as MPI and SHMEM.
Alternatively, a remote peer can be referenced using a provider specified
address, an option that allows data transfer calls to pass encoded
addressing data directly to the hardware,
avoiding additional memory reads during the data transmission.

\subsection{Completion Services}
Libfabric exports asynchronous interfaces,
and completion services are used to report
the results of previously initiated asynchronous operations.
Completions may be reported either by using event queues, which
provide details about the operation that completed,
or by lower-impact counters, which simply return the number
of operations that have completed.

Applications select the type of completion structure.
Different completion structures provide varying amounts of
information about a completed request.
This allows for creating compact arrays of completion structures,
and ensures that data fields that would otherwise be ignored by an application
are not filled in by a provider.

\subsection{Data Transfer Services}
These services are sets of interfaces designed around different
communication paradigms.
Figure~\ref{fig:architecture}
shows four basic data transfer interface sets,
plus
triggered operations
that are strongly related to the data transfer operations.

(i) Message queues expose the ability to send and receive data
in which message boundaries are maintained.
They act as FIFOs, with sent messages matched with receive
requests in the order that messages are received.

(ii)Tag matching is similar to message queues in that it maintains
message boundaries, but differs in that received messages
are directed to receive requests based on small steering tags
that are carried in the sent message.

(iii) RMA stands for ``Remote Memory Access''.
RMA transfers allow an application to write data from local memory
directly into a specified memory location in a target process,
or to read data into local memory directly from a specified memory location
in a target process.

(iv) Atomic operations are similar to RMA transfers in that they allow
direct access to a specified memory location in a target process,
but the differ in that they allow for manipulation of the value found
in that memory, such as incrementing or decrementing it.

Different data transfer interfaces are defined in order to eliminate
branches that would occur in the provider if only a single, more general
function call were defined.
This eliminates checks inside the provider, enabling it to
preformat command buffers to further reduce the number of instructions
executed in a transfer.

