The Windows OpenFabrics (WinOF) release package contains the following:
OpenFabrics InfiniBand core drivers and Upper Level Protocols (ULPs):
-
HCA
Drivers -
Mellanox
InfiniHost & ConnectX low level drivers; See list of
supported devices below
-
Infiniband Core components
-
Upper Layer Protocols:
IPoIB, WSD, VNIC, SRP Initiator and DAT/DAPL
OpenFabrics utilities:
Documentation
-
User's manual
-
Release Notes
<Return-to-Top>
CPU architectures
-
x64 (x86_64, amd64)
-
x86
-
IA64
Operating Systems
Supported HCAs (Host Channel Adapters)
Mellanox
-
MT23108 - InfiniHost
-
MT25218 InfiniHost III Ex, memfree
-
MT25204 InfiniHost III Lx
-
MT25408 -
ConnectX DDR Channel Adapter, not
yet available on IA64 platforms.
Both SDR and DDR mode of the InfiniHost III family are supported.
For official Firmware (FW) versions and update tools please see:
http://www.mellanox.com/support/firmware_table.php
Current HCA firmware version can be viewed from the 'vstat' command.
Supported Switches
<Return-to-Top>
User mode tools and diagnostics are installed in '%SystemDrive%\Program
Files (x86)\WinOF'. Although driver modules initially reside in
'%SystemDrive%\Program Files (x86)\WinOF', when installed the Windows device
installer relocates driver modules to %SystemRoot%\system32 & %SystemRoot%\SysWOW64'.
DAT & DAPL runtime libraries are installed into %SystemRoot% in order to be
accessible to user mode applications.
WinOF user mode executables are designed to be run from
a Command Prompt window; see 'Start->Program Files-> Windows OpenFabrics-> Command Prompt'.
'%SystemDrive%\Program Files (x86)\WinOF' is appended to the system wide search
path; new command windows created after the installation will have the updated
%PATH% environment variable, hence will be able to access WinOF utilities.
The default installation installs released/free (not
checked/debug) versions of drivers and executables.
A single instance of a subnet manager, opensm.exe, must be
running on a fabric connected node in order for the Infiniband fabric to be
configured and useful; either Windows or Linux opensm work well.
It is recommended that OpenSM be run as a Windows service. A typical
WinOF install will install OpenSM as a Windows service which
is disabled; after an installation, choose your openSM node and start the openSM
service.
See the WinOF Manual for details on opensm as a service.
By selecting the 'OpenSM_service_enabled' feature, an OpenSM subnet
management service will be automatically started.
Note, the opensm.exe process must continue to run in order
to maintain Infiniband fabric configuration. Should the opensm.exe process die,
restart the service if not automatic or re-run opensm.exe in order to continue
correct fabric operation.
For more subnet management information, consult the Windows
OpenFabrics manual.
<Return-to-Top>
Requirements
Install HCA hardware prior to installing the Windows OpenFabrics Release
package. Upon reboot, cancel any attempts to install new device hardware from
the "Found New Hardware Wizard".
If you have previously installed openib-windows, Windows OpenFabrics
packages or vendor supplied Infiniband packages, uninstall and reboot prior to
installing this version of Windows OpenFabrics (WinOF).
For an existing Windows OpenFabrics uninstall
Programs-> Windows OpenFabrics-> Uninstall WinOF
-or-
Control Panel -> Add or Remove Programs-> Windows
OpenFabrics->Remove.
openib-windows uninstall:
- Stop the openSM subnet manager if it is running on the
local node.
- Disable any Local Area Connections which are bound to the
IPoIB network adapter.
- Uninstall the IPoIB driver - My
computer->Manage->Devices->Network Adapters->IPoIB*
- Uninstall the Infiniband HCA device; don't forget the
System Device-> InfiniBand Fabric
REBOOT
Upon system restart, cancel the 'Found New Hardware Wizard' attempt to install
drivers for newly discovered PCI device.
'My Computer->Manage->Device Manager' should display
'Other Devices->?PCI device' which is your Infiniband HCA device.
Install
From a login with administrator privileges, double-click the
WinOF_M-m_arch_setup.msi file to begin the Windows OpenFabrics installation;
where '_arch_' can be (_x64_ for Intel EMT64 systems or amd64, _x86_ for x86
compatible systems or _ia64_ for Intel IA64 systems).
Follow the instructions.
Answer 'yes' or 'Continue' to any notifier windows which reference non-signed
driver installation. These non-signed driver install questions can be disabled
via 'My Computer->Properties->Hardware->Driver Signing->Ignore'.
Bottom-line - WinOF drivers in this package are not WHQL'ed although they are
built from driver source which has been WHQL'ed.
**** WARNING ****
Should any 'Welcome to the Found New Hardware Wizard' windows popup after the
WinOF install has started, just ignore (window will disappear in a few
seconds) or 'Cancel' the errant popup window.
For Windows Server 2003 (SP1) you can safely ignore the errant FNHW popup
window; for Windows XP (SP2) the errant FNHW popups need to be 'cancelled'
in order for the WinOF install to proceed. The report back from Microsoft on these 'Found New Hardware Wizard' windows
popping up is they are due to the fact the WinOF drivers are not Digitally
signed (non-WHQL'ed) and hence are not trusted by Windows. Even though the
Ignore warning on installation of non-signed drivers is enabled (My
Computer->Properties->Hardware->Driver Signing->Ignore) the FNHW is still
invoked. MS claims this is a 'feature'.
An artifact of cancelling an errant 'Found New Hardware' popup window may
induce an errant taskbar Information notifier/bubble indicating there may
have been problems installing your hardware drivers;
IGNORE/cancel this information bubble, likely the device installation
proceeded without error. Check the Device Manager views for the
Infiniband HCA and IPoIB Network Adapters for initial verification of
correct installation.
You do need to answer 'Yes' or 'Continue' to those notifier windows which
reference non-signed driver installation. If you have previously
set My Computer->Properties->Hardware->Driver Signing->Ignore, then you will
NOT be queried about installation of non-signed drivers; although the FNHW
windows likely will occur.
Should the install appear to hang, look around for occluded notifier windows
waiting for a response which
are covered by existing windows.
Install type: Custom only
- InfiniBand Core:
- HCA driver - Mellanox: InfiniHost or ConnectX
- IB core stack and libraries.
- Release Notes, Manual
- Subnet Management [opensm], Performance & Diagnostic tools
- Optional:
IPoIB - Internet Protocols over
InfiniBand
Winsock Direct (Win2K3 only)
VNIC - Virtual Ethernet device over Infiniband
SRP - SCSI over RDMA Protocol
DAT/DAPL
- RDMA device independent, user-mode Direct Access Transport & Direct Access
Provider
- DAT v1.1 runtime libraries.
- DAT v1.1 application build environment
(header files & link files).
- DAT/DAPL v2.0 runtime libraries
- DAT v2.0 application build environment
OpenSM_service_started - InfiniBand Subnet Management
enabled and started as a Windows Service.
By default OpenSM is installed as a disabled Windows Service.
- The 'default' installation includes IB core, IPoIB, WSD (Win2K3 only),
DAT 1.1 runtime libraries, OpenSM service (disabled).
<Return-to-Top>
To perform a silent/unattended installation, invoke the following command
from a command window.
start/wait msiexec.exe /i WinOF_x64_1-0-1.msi /quiet /passive /log
%TEMP%\WOF_install.log
'/log dev:path\logFilename.log' is optional.
msiexec.exe /? for all options.
An unattended install will install the following 'default' install options:
- Infiniband Core - HCA driver, IB core stack and libraries,
Documentation
- IPoIB - Internet Protocols over InfiniBand
- WSD - (Win2k3 only, not installed on XP) Win Sock Direct
- DAT & DAPL (v1.1) runtime libraries.
Should WSD not be a desirable unattended install option (Win2K3 only),
currently you would install unattended then execute the command 'installsp -r'
on each node to remove WSD.
See Unattended
Installations for further references.
<Return-to-Top>
The Windows command 'netsh' will assist in assigning an IP address to an
IPoIB interface.
By default, an IPoIB interface is configured to use DHCP. The following
discussion deals with setting a static IP address on an IPoIB interface.
Display all network interface configuration information
netsh interface ip show config
netsh interface ip show address
"Local Area Connection 3"
Config a static IP address for an IPoIB interface: [Local Area Connection 3
== 1st IPoIB interface], 10.10.4.23 is the assigned IP address
with a netmask of 255.255.255.0, the gateway is 10.10.4.1 metric is 2.
netsh interface ip set address
"Local Area Connection 3" static 10.10.4.23 255.255.255.0 10.10.4.1 2
See netsh for further
reference.
<Return-to-Top>
To uninstall a WinOF package from a single node
- Programs-> Windows OpenFabrics-> Uninstall WinOF
- Control Panel-> Add Remove Programs-> Windows OpenFabrics-> Remove
Unattended
Uninstall
From a Command Window invoke the following command; a system reboot will
occur after the uninstall.
start/wait msiexec.exe /x {3A6A276E-A636-48E8-8624-9A0DE3F2A15E} /quiet /passive
/forcerestart
- or -
start/wait msiexec.exe /x WinOF_x64_1-0-1.msi /quiet /passive /forcerestart
# WinOF_xxxx.msi is the file used to install WinOF.
msiexec.exe /help for details.
'/log dev:path\logfilename.log' if you wish to later view the install
results.
If for some reason the Product Code {GUID} changes, the latest code can
be located via
Programs-> Windows OpenFabrics-> Uninstall WinOF<right-click>Properties
Infrequently, the 'System Device->Infiniband Fabric' may not correctly uninstall. Make sure the Infiniband Fabric device is
completely uninstalled; use device manager for checking. The effect of a
partially uninstalled Infiniband Fabric device is IPoIB load failure
upon the next WinOF install.
<Return-to-Top>
Down rev firmware will result in IPoIB not installing. Consult the event
viewer, system error records, look for 'mthca' entries generated when the HCA
driver loads. One of the mthca entries will display the current firmware
revision.
Current Mellanox Firmware Revisions Matrix
MT23108
(InfiniHost)
§Minimal FW version: 3.3.2
§Optimal FW version: 3.4.0
MT25208
(InfiniHost III Ex, Tavor mode)
§Minimal FW version: 4.7.0
§Optimal FW version: 4.7.400
MT25218
(InfiniHost III Ex, memfree)
§Minimal FW version: 5.1.0
§Optimal FW version: 5.1.400
MT25204
(InfiniHost III Lx)
§Minimal FW version: 1.0.800
§Optimal FW version: 1.1.0
If
HCA FW is older than minimal FW
Error
reported to system event log
HCA loads as memory controller to allow FW upgrade
If HCA
FW is equal or newer than optimal version
Else
HCA loads OK.
Warning will be issued to system event log
My computer-> Manage-> Event Viewer-> System Events
Firmware updates
are available at
www.mellanox.com/support/firmware_download.php
Incomplete Previous Uninstall
Should the previous uninstall silently fail, the next installation
can fail in the Driver Install phase with the following error:
DriverInstall - HCA Ret 2 Error 0
At this juncture, check the Device Manager and uninstall the PCI
device from 'InfiniBand Host Channel Adapters', and make sure you uninstall the system device
'InfiniBand Fabric'.
REBOOT and retry the WinOF installation again.
From the Device Manager you should find the following devices:
- InfiniBand Host Channel Adapters -> InfiniHost
- System Devices -> InfiniBand Fabric
- Network Adapters -> OpenIB IPoIB Adapter (an Instance per HCA port).
Validation Test
Open a WinOF command window
- Start->Program Files->Windows OpenFabrics -> Command
Window
- run vstat.exe to view HCA configuration.
- If IPoIB is installed (Device Manager->Network Adapters), ping another node on the InfiniBand fabric 'ping a.b.c.d'
Limit the number of Subnet Managers on your fabric; one SM per fabric is
sufficient, although redundant Subnet Managers are supported.
- After a 'typical' WinOF install, OpenSM is installed as a disabled
Windows service named "InfiniBand Subnet Management". To start the OpenSM
service, use My computer->Manage->Services and Applications->Services->InfiniBand Subnet Manager->Start.
Set the service properties to 'auto' in order to restart upon the next
system reboot.
- Install OpenSM as an automatic/running Windows service:
a) Request a 'Custom' install, selecting the 'OpenSM_service_enabled' feature. Once
the install has completed, check that the Infiniband Subnet Manager service is
running: My computer->Manage->Services and Applications->Services->InfiniBand Subnet
Manager.
- Consult the OpenSM log file @ %SystemRoot%\temp\osm.log to see what
OpenSM thinks is happening.
<Return-to-Top>
Local Network Adapter ordering, which results in the IP address advertised
for the system, can be managed from
'My Network Places-> Advanced Tab-> Advanced settings'
From the Advanced settings display, the ordering
of Local Network Adapters can be
reordered after the IPoIB Local Network Adapters have been installed. Please
check for your desired adapter ordering.
<Return-to-Top>
- Add/Remove Programs entry identifies the WinOF release version.
- Start->Programs->Windows OpenFabrics->Release Notes
(2nd Line)
- Driver confirmation:
Query My Computer-> Manage-> Device Manager-> Network
Adapters-> OpenIB IPoIB Adapter-> Properties-> Driver-> Details
The subversion (svn) revision number will be listed under
'Driver Version'. The svn revision number will match what's listed in the
release notes.
- Hold the mouse point over \Program Files(x86)\WinOF\ib_read_bw.exe to display
the file version; the 4th field is the svn revision number.
<Return-to-Top>
The QLogic VNIC
(Virtual Network Interface Card) driver in conjunction with the QLogic Ethernet
Virtual I/O Controller (EVIC) provides virtual Ethernet interfaces and transport
for Ethernet packets over Infiniband.
Users can modify NIC parameters through User Interface icon in Network
Connections:
( Properties->"Configure..." button -> "Advanced" Tab).
Parameters
available:
Vlan Id (802.1Q)
values from 0 to
4094 ( default 0, disabled )
This specifies if VLAN ID-marked packet transmission is enabled and, if so,
specifies the ID.
Priority (802.1P)
values from 0 to 7
( default 0, feature disabled)
This specifies if priority-marked packet transmission is enabled.
Payload MTU size
values from 1500
to 9500 (default 1500)
This specifies the maximum transfer unit size in 100 bytes increments.
Recv ChkSum offload
(default enabled)
This specifies if IP protocols checksum calculations for receive is offloaded.
Send ChkSum offload
(default enabled)
This specifies if IP protocols checksum calculations for send is offloaded.
Secondary Path
(default
disabled)
Enabled - If more than one IB path to IOC exist then secondary IB instance of
virtual port will be created and configured with the same parameters as primary
one. Failover from Primary to Secondary IB path is transparent for user
application sending data through associated NIC.
Disabled only one path at a time is allowed. If more than one path to IOC
exists then failed path will be destroyed and next available path will be used
for new connection. With this scenario there is a possibility new interface
instance will be assigned different MAC address when other hosts compete for
EVIC resources.
LBFO Bundle Id
(default disabled) Enabling support for OS provided Load Balancing and Fail
Over functionality on adapter level.
If enabled group ID can be selected from predefined names.
Heartbeat interval
configures
interval for VNIC protocol heartbeat messages in milliseconds.
0 heartbeats disabled.
Note:
To take advantage of the features supported by these options, ensure that the
Ethernet gateway is also configured appropriately. For example, if the Payload
MTU for a VNIC interface is set to 4000,
the MTU at the EVIC module must also be set at least 4000
for the setting to take effect.
<Return-to-Top>
In order for DAT/DAPL programs to execute correctly, the 'dat.dll'
file must be present in the current directory,
%SystemRoot% or in the library search path.
WinOF installation places the dat.dll and dapl.dll
files in the '%SystemRoot%'
folder (a.k.a. \Windows).
The DAT/DAPL configuration file by convention is defined as
'%SystemDrive%\DAT\dat.conf'. If the file '%SystemDrive%\DAT\dat.conf' does not exist,
a template configuration file will be installed as '%SystemDrive%\DAT\dat.conf'.
In order to preserve existing installations, the dat.conf file is not
automatically installed if '%SystemDrive%\DAT\dat.conf' exists.
A sample dat.conf file is always installed as '\Program Files (x86)\WinOF\dat.conf
'.
The default DAT configuration file specification can be overridden by use
of the environment variable DAT_OVERRIDE. Such that 'DAT_OVERRIDE=D:\MyDAT\test.conf'
would be a valid override specification.
Within the dat.conf file, the DAPL library specification can be
located as the 5th whitespace separated line argument. By default the DAPL
library file is installed as %WINDIR%\dapl.dll'.
Should you choose to relocate the DAPL library file to a folder
where whitespace appears in the full library path specification, then the full
library file specification must be contained within double-quotes. A side effect
of the double-quotes is the library specification is treated as a Windows string
which implies the '\' (backslash character) is treated as an 'escape' character.
Hence all backslashes in the library path must be duplicated when enclosed in
double-quotes (e.g., "C:\\Programs Files (x86)\\WinOF\\dapl.dll").
DAT/DAPL version 2.0 runtime
libraries are identified as dat2.dll and dapl2.dll, both in %SystemRoot%; see
manual for further details.
<Return-to-Top>
The WinOF stack does not install the SRP driver by default.
If SRP is
selected in the custom installation window, it will only be copied during WinOF
installation. To complete the SRP driver installation, an SRP target must be
detected; a Subnet Manager must be running somewhere in the InfiniBand subnet.
Upon the detection of an SRP target, the "New Hardware Found" Wizard pops up.
- Select Install Automatically and click Next. This installs the I/O unit
device.
Once completed, the "New Hardware Found" Wizard pops up again.
- Select Install Automatically and click Next. This installs the SRP driver.
<Return-to-Top>
