OpenFabrics Enterprise Distribution for Windows

winOFED 3.2

Release Notes

06/25/2013

New Features, Items and Behaviors

Overview

Supported Platforms, Operating Systems and InfiniBand Hardware

Installation Notes

How to Install

Unattended Install

HCA Device Driver Installation

Server 2008 HPC Install Notes

Setting the IPoIB Interface IP Address

Uninstall

Verifying the Installation

Trouble Shooting

InfiniBand Subnet Management as a Windows Service

Local Network Adapter Ordering

Which release is installed?

QLogic VNIC Configuration

DAT & uDAPL Configuration

SRP (SCSI RDMA Protocol) Driver Installation

WinVerbs

Known Issues


New Items, Features and/or Behaviors

 

Overview

The OpenFabrics Enterprise Distribution release package is composed of software modules intended for use on Microsoft Windows based computer systems connected via an InfiniBand fabric.

 Binary files generated from the OpenFabrics Alliance (OFA) windows developers subversion (svn) source tree 'http://sofa.openfabrics.org/svn/ofw' (branches\WOF3-2 svn revision 3635 are packaged into a WIX (Windows Installer Xml) single file install package referred to as the OpenFabrics Enterprise Distribution for Windows release 3.2.
 

The winOFED 3.2 release is a full featured release as it contains:

The OpenFabrics Enterprise Distribution for Windows (winOFED) release package contains the following:

OpenFabrics InfiniBand core drivers and Upper Level Protocols (ULPs):

OpenFabrics utilities:

Documentation

<Return-to-Top>

 

Supported Platforms, Operating Systems and Infiniband Hardware

CPU architectures

Operating Systems

Supported HCAs (Host Channel Adapters)

Mellanox - Connect-X[1-4] InfiniBand HCA products are supported; InfiniHost HCAs are no longer supported.

Mellanox Firmware (FW) versions and update tools

Current HCA firmware version can be viewed from the 'vstat' command or the Mellanox firmware tool flint.

How to find the PSID of a Mellanox HCA

From a winOFED installed system: vstat.exe

Should the OFED stack not install; possibly due to outdated firmware:

Download and install the Mellanox Firmware burning tools from Mellanox Support

After you determine the PSID,

Download Mx Firmware image .zip file from Mellanox Firmware download

How to find the PSID of the Mellanox card if the OFED stack did not install, otherwise use vstat.

mst status
Found 2 devices:
mt26418_pciconf0
mt26418_pci_cr0

flint -d mt26418_pci_cr0 query
In the output, you will find the PSID.

Supported Switches

<Return-to-Top>

 

Installation Notes

User mode tools and diagnostics are installed in '%SystemDrive%\Program Files\OFED'.  Although device driver modules initially reside in '%SystemDrive%\Program Files\OFED\Drivers', during installation the Windows device installer copies driver files to %SystemRoot%\system32\drivers, %SystemRoot%\system32 & %SystemRoot%\SysWOW64.
DAT & DAPL runtime libraries are installed into %SystemRoot%\system32 &  %SystemRoot%\SysWOW64 in order to be accessible by user mode applications.
OFED user mode executables are designed to be run from a Command Prompt window; see  'Start->Program Files->OpenFabrics Enterprise Distribution-> OFED Command Window'.

'%SystemDrive%\Program Files\OFED' is appended to the system wide search path environment variable 'PATH'; any command windows created after the installation will have the updated %PATH% environment variable, hence will be able to access OFED executables.

The ‘default’ installation installs ‘released/free’ (not checked/debug) versions of drivers and executables. Debug/Checked versions are available as an install feature.

A single instance of a subnet manager, opensm.exe, must be running on a fabric connected node in order for the Infiniband fabric to be configured and useful; either Windows or Linux versions of openSM work well.

It is recommended that OpenSM be run as a Windows service.  A typical OFED install will install OpenSM as a Windows service which is disabled; after an installation, choose your openSM node and start the openSM service.  See the OFED Manual for details on opensm as a service.
By selecting the 'OpenSM_Service_Started' install feature, a local OpenSM subnet management service will be automatically started.

Note, the opensm.exe process must continue to run in order to maintain an Infiniband fabric configuration. Should the opensm.exe process die, restart the service if not automatic or re-run opensm.exe in order to continue correct fabric operation.

For more subnet management information, consult the Windows OpenFabrics Enterprise Distribution manual.

<Return-to-Top>

 

How to Install

Summary

Requirements

Install HCA hardware prior to installing the Windows OpenFabrics Release package. Upon reboot, cancel any attempts to install new device hardware from the "Found New Hardware Wizard".

If you have previously installed openib-windows, WinOF (Windows OpenFabrics), winOFED packages or vendor supplied Infiniband packages, uninstall and reboot prior to installing this version of OpenFabrics Enterprise Distribution for Windows.

For an existing Windows OpenFabrics Enterprise Distribution (OFED) uninstall
    Start-> Programs-> OpenFabrics Enterprise Distribution-> Uninstall OFED
      -or-
    Control Panel -> Add or Remove Programs->OpenFabrics Enterprise Distribution->Uninstall.

openib-windows (IB stack before WinOF 1.0 release) uninstall: (not the normal uninstall case)

  1. Disable any Local Area Connections which are bound to the IPoIB network adapter.
  2. Uninstall the IPoIB driver - My computer->Manage->Devices->Network Adapters->IPoIB*
  3. Stop the openSM subnet manager if it is running on the local node.
  4. Uninstall the Infiniband HCA device; don't forget the System Device-> InfiniBand Fabric

 

REBOOT

Upon system restart, cancel the 'Found New Hardware Wizard' attempts to install drivers for newly discovered PCI device (HCA).

'My Computer->Manage->Device Manager' should display 'Other Devices->?PCI device' which is your Infiniband HCA device.

Install

For all supported Windows variants, double-clicking the installer file (.msi) works correctly to start an OFED installation.
From a login session with administrator privileges, File Explorer view, double-click the OFED_M-m{-p}_OSE_arch.msi file to begin the Windows OpenFabrics Enterprise Distribution installation.

Where 'M-n{-p}' indicates Major release number, Minor release number, and optional point-release number.

 OSE - Operating System Environment:

'_arch_' will be one of

Follow the on-screen instructions.

Answer 'yes' or 'Continue' to any notifier windows which reference non-WHQL'ed driver installation.
These non-WHQL'ed driver install questions can not be disabled via 'My Computer->Properties->Hardware->Driver Signing->Ignore'.

Bottom-line - The digitally signed OFED drivers in this package are not WHQL'ed per se, although they are built from driver source which has been or is in process of being WHQL'ed by hardware vendors who participate in the open source OFED for Windows development process.

 

 **** WARNING ****

You do need to answer 'Yes' or 'Continue' to those notifier windows which reference non-WHQL driver installation. If you have previously set My Computer->Properties->Hardware->Driver Signing->Ignore, the FNHW windows likely will occur anyway.

Should the install appear to hang, look around for occluded notifier windows waiting for a response which are covered by existing windows.
 

Install type: Custom only

<Return-to-Top>

 

HCA Driver Installation

OFED driver installation uses the Device Installation Frameworks for Applications (DIFxApp) and the Windows Driver Store.
The  Microsoft Plug-n-Play (PNP) subsystem controls which Infiniband HCA (Host Channel Adapter) device driver is loaded from the Windows Driver Store.  HCA driver selection is based on PNP device ID probing, thus InfiniHost, ConnectX or both drivers can be loaded by the PNP subsystem.
Using DIFxApp and PNP device probing results in the OFED installation not requiring a specific HCA model [ConnectX] feature selection; HCA vendor is the feature selection criteria.


 

Unattended Install

An unattended OFED install will install the following 'default' options
  • Infiniband Core modules - Mellanox HCA driver (see HCA driver selection note), IB core stack, libraries, utilities and documentation.
  • IPoIB - Internet Protocols over InfiniBand
  • WSD - (installed although not enabled: Win7 Server2K8/R2) Win Sock Direct
  • NDv2 - NetworkDirect [started automatically (via ndinstall.exe) only for Server 2008/HPC].
  • DAT & DAPL (v2.0) runtime libraries + dapltest2.exe (dt-svr & dt-cli).

HCA selection for all Windows variants (Windows 7, Server 2008 R2, HPC)

Default HCA driver selection is driven by the Windows PNP subsystem; no user selection required or supported.

Should WSD be a desirable unattended install option (Win7/2008 R2 only), currently you would install unattended then execute the
command 'installsp -i' on each node to enable WSD.

To perform a silent unattended installation for Server 2008 R2, invoke the following command from a command window.

start/wait msiexec.exe /i OFED_3-2_win7_x64.msi  /qn  /quiet  /log %TEMP%\OFED-install.log

'/log dev:path\logFilename.log' is optional.

msiexec.exe /? for all options.

Examples:

Server 2008 - Mellanox Connect-X HCA Hardware


Non-HCA Install Features added to above msiexec command line examples:

Default install +  OpenSM Started on the local system

Default install + SRP client

<Return-to-Top>

 

Server 2008 R2 HPC Install Notes

Device drivers in OFED 2.0 and later releases are Digitally signed by the 3rd party Software Publisher 'OpenFabrics Alliance', although they are not WHQL'ed (Windows Quality Hardware Labs) certified. The OFED source code is used  by hardware vendors to achieve WHQL certification.

HPC compute node provisioning

Microsoft has chosen WDM to provision HPC cluster compute nodes; see  Microsoft HPC installs for details.
Using the HPC Cluster Manager tool, OFED device drivers can be included in the compute node provisioning process by adding the device driver's .inf file to the operating system image.

In support of the new Microsoft WDM HPC cluster provisioning process, OFED install suggests one of two possible paths:

  1. Install winOFED on the head node and then add compute node drivers (HCA, IPoIB) from the head node (\Program Files\OFED\Drivers) to the Windows operating system image via the cluster manager toool (Configuration->Images->Manage Drivers).
     
  2. OFED now supports an administrative install mode which is a method to extract OFED files to the local filesystem without actually installing them; thus making OFED files accessible to WDM provisioning.

    msiexec /A OFED_2-1_wlh_x64.msi TARGETDIR=dev:\path-to-extracted-files    # last folder in path-to-extracted-files must NOT exist.

    example:  msiexec /A OFED_3-2_wlh_x64.msi TARGETDIR=%TEMP%\OFED   (note: TARGETDIR is case sensitive, OFED must not exist)
    The above command creates the folder structure: %TEMP%\OFED\PFiles\OFED\

    Once the administrative install has completed, add the required winOFED device drivers to the Compute node operating system image via the Cluster Manager tool.

OFED device driver-less installs:

Normally OFED device driver files are installed by processing the driver '.inf' files. To install the OFED package by selecting install features without installing any device drivers [HCA, IPoIB, ND, WSD, SRP or VNIC] (assumed reliance on WDM to install device drivers) can be accomplished with the following interactive install command:

start/wait msiexec /I OFED_3-2_wlh_x64.msi  NODRV=1

To skip driver installation for an unattended install

    start/wait msiexec /I OFED_3-2_wlh_x64.msi /qn /quiet NODRV=1

Note: when uninstalling OFED from an installation which was done with NODRV=1, you MUST include NODRV=1 on the uninstall command line; otherwise OFED drivers will be uninstalled also.

    start/wait msiexec /X OFED_3-2_wlh_x64.msi /qn /quiet NODRV=1
 

Unattended winOFED installation from the .msi file in the HPC Environment

The lack of winOFED WHQL driver certification places winOFED drivers in the 'unknown 3rd party SW publishers' category.
When installing 'Unknown 3rd party SW publishers' drivers, Windows prompts on the install console for instructions as what to do (Install, Trust or not)?
This prompting can result in a failed remote node install when performing a first-time unattended OFED install from the .msi file.
By preloading the remote node's Trusted 3rd party Software Publisher Certificate Store, the unattended install will proceed without prompting, thus allowing the OFED install to complete successfully.

Be aware: if a node is reimaged (same or different template applied) the established trust of the OpenFabric Alliance is destroyed, thus the OFA certificate needs to be reinstalled.

How to install OFED drivers and components from the winOFED .msi file in the HPC Server 2008 environment.

Common HPC compute node install steps

<Return-to-Top>

 

Setting the IPoIB Interface IP Address

The Windows command 'netsh' will assist in assigning an IP address to an IPoIB interface.
By default, an IPoIB interface is configured to use DHCP. The following discussion deals with setting a static IP address on an IPoIB interface.
If your system has two Ethernet ports, then IPoIB interfaces (HCA port 1, 2...) will be assigned to 'Local Area Connection 3' and 'Local Area Connection 4'.  If you system has a single Ethernet port, then IPoIB interface is 'Local Area Connection 2'.

For Server 2008/HPC systems, the OFED installation follows the above discussion. In defining HPC network interfaces, the IPoIB network will eventually be identified as the 'Application' network '.  A few minutes after a compute node is installed, the HPC  network manager will rename the installed IPoIB 'Local Area Connection X' network interface to the 'Application' network.

Display all network interface configuration information
        netsh interface ip show config
        netsh interface ip show address "Local Area Connection 3"

Config a static IP address for an IPoIB interface: [Local Area Connection 3 == 1st IPoIB interface], 10.10.4.23 is the assigned IP address
with a netmask of 255.255.255.0, the gateway is 10.10.4.1 metric is 2.

        netsh interface ip set address "Local Area Connection 3" static 10.10.4.23 255.255.255.0 10.10.4.1 2

See netsh for further reference.

<Return-to-Top>

 

Uninstall

To uninstall a OFED package from a single node

  1. Start--> All Programs-> Windows OpenFabrics-> Uninstall OFED
  2. Control Panel-> Add Remove Programs-> Windows OpenFabrics-> Remove

Unattended Uninstall

From a Command Window invoke the following command; a system reboot will occur after the uninstall.

start/wait msiexec.exe  /x {3A6A276E-A636-48E8-8624-9A0DE3F2A15E} /quiet /forcerestart
  - or -
start/wait msiexec.exe  /x OFED_3-2_win7_x64.msi /quiet /forcerestart        # OFED_xxxx.msi is the file used to install OFED.

msiexec.exe /help for details.

'/log dev:path\logfilename.log'  if you wish to later view the install results.

If for some reason the Product Code {GUID} changes, the latest code can be located via
Programs-> Windows OpenFabrics-> Uninstall OFED<right-click>Properties

Infrequently, the 'System Device->Infiniband Fabric' may not correctly uninstall. Make sure the Infiniband Fabric device is completely uninstalled; use device manager for checking. The effect of a partially uninstalled Infiniband Fabric device is IPoIB load failure upon the next OFED install.

Uninstall Notes

  1. If the SRP (SCSI RDMA Protocol) driver has been previously installed, then in order to achieve a 'clean' uninstall, the SRP target drive(s) must be released.
     
    • Use the following diskpart.exe commands from an administrator privileged command window:
    • c:\windows\temp> diskpart.exe
          SELECT VOLUME=<Drive Letter of the SRP target to be released>        # example: SELECT VOLUME=L
          OFFLINE DISK
          EXIT
    • For additional SRP drives, repeat SELECT & OFFLINE DISK with new Drive Letter.

    The consequences of not releasing the SRP target drive(s) is that after the uninstall reboot there are lingering InfiniBand driver files. These driver files remain because while the SRP target is active they have references, thus when the uninstall attempts to delete the files the operation fails.

<Return-to-Top>

 

Verifying the OFED install

<Return-to-Top>

 

Trouble Shooting

From a command window type the following quick-start commands:

To create a LOG file of your OFED installation, execute this command: msiexec /I OFED_xxx_yyy.msi  /Lv %TEMP%\OFED-install.log
View %TEMP%\OFED-install.log

msiexec /? for details.

Server 2008 R2 or WIndows-7 can also fail to install due to driver install files left over from previously failed install attempts; see %windir%\system32\DriverStore\{Temp|FileRepository}. Look for folders with the following in the folder name: mthca, mlx4_, netipoib, srp, vnic. Once you have located one of these folders you need to remove it as win2k8 R2 will use these files instead of what your attempting to install.; removing the folders is a pain.
Right-click the folder and select Properties->Security->Advanced, select owner tab, select Edit button, select Administrator and check the 'Replace owner on subcontainers and objects', now click 'OK', OK again, select Edit button from the securities tab, check allow 'Full Control' box and click OK, OK again. At this point you should be back to the file explorer. Now you can delete the folder!

In emergency situations (as administrator) run IBcleanup.bat; download from https://www.openfabrics.org/downloads/Windows/etc/IBcleanup.bat

Setupapi.log has moved in Server 2008, see '%windir%\inf\setupapi-dev.log'. Setupapi-*.log can at times contain useful install debug info.

Server 2008 R2/HPC all use dpinst.exe to preinstall drivers into the driver store. Windows PNP (Plug-n-Play) later will demand load drivers when the hardware is recognized. dpinst.exe writes it's error information to '%windir%\dpinst.log'.

Down rev firmware will result in IPoIB not installing or other anonmolus behavior. Consult the event viewer, system error records, look for 'mthca' or 'mlx4*' entries generated when the HCA driver loads. One of the mthca or mlx4 entries will display the current firmware revision.

Firmware version information and updates are available at www.mellanox.com/support/firmware_download.php

•
–
•If HCA FW is older than minimal FW
Error reported to system event log
–HCA loads as memory controller to allow FW upgrade
•If HCA  FW is equal or newer than optimal version
 
–HCA starts OK.
•Else
HCA loads OK.
–Warning will be issued to system event log
–
My computer-> Manage-> Event Viewer-> System Events; search for mthca, mlx4 or ipoib entries.

ttcp or other network applications fail to connect?
    Make sure the firewall settings allow said program or TCP port to pass.

<Return-to-Top>

 

Incomplete Previous Uninstall

Should the previous uninstall silently fail, the next installation can fail in the Driver Install phase with the following error:
    DriverInstall - HCA Ret 2 Error 0

At this juncture, check the Device Manager and uninstall the PCI device from 'InfiniBand Host Channel Adapters', and make sure you uninstall the system device 'InfiniBand Fabric' if it is present.

REBOOT and retry the OFED installation again.

See ibscan.bat and ibcleanup.bat @ http://www.openfabrics.org/downloads/windows/etc/.
ibscan will display OFED files on your system.
ibcleanup will attempt to remove all OFED files from your syste. WARNING - ibcleanup is not valid as a OFED uninstall tool!

<Return-to-Top>

 

Correct Installation Validation

From the Device Manager you should find the following devices:

Validation Test

Open a OFED command window

  1. Start->Program Files->Windows OpenFabrics -> Command Window
  2. run vstat.exe to view HCA configuration.
  3. If IPoIB is installed (Device Manager->Network Adapters), ping another node on the InfiniBand fabric 'ping a.b.c.d'
     

<Return-to-Top>

 

Subnet Management as a Windows Service

Limit the number of Subnet Managers on your fabric; one SM per fabric is sufficient, although redundant Subnet Managers are supported.

  1. After a 'typical' OFED install, OpenSM is installed as a disabled Windows service named "InfiniBand Subnet Management".
    To start the OpenSM service, from a command window type 'services.msc' to start the Services manager.  From the Services manager, find and select 'InfiniBand Subnet Manager' view; Click the start button and set the service properties to 'auto' in order to restart on the next system reboot.
     
  2. Install OpenSM as an automatic/running Windows service:
    a) Select the 'OpenSM_service_Started' install feature. Once the install has completed, check that the Infiniband Subnet Manager service is running: From a command windows type 'services.msc' to start the Services manager. Find and open the InfiniBand Subnet Manager view.
     
  3. Consult the OpenSM log file @ %SystemRoot%\temp\osm.log to see what OpenSM thinks is happening.

<Return-to-Top>

 

Local Network Adapter Ordering

Local Network Adapter ordering, which results in the IP address advertised for the system, can be managed from
    'My Network Places-> Advanced Tab-> Advanced settings'
From the Advanced settings display, the ordering of Local Network Adapters can be reordered after the IPoIB Local Network Adapters have been installed. Please check for your desired adapter ordering.

<Return-to-Top>

 

Which OFED release is installed?

  1. Start->Programs->OpenFabrics Enterprise Distribution->About OFED.
    Displays a window containing winOFED release information.
     
  2. Hold the mouse point over \Program Files\OFED\ib_read_bw.exe to display the file version; [OFED-Major.Minor.not-used.svn revision number].
    If a file has been built 'checked' then a '(DEBUG)' tag will be displayed.

<Return-to-Top>

 

QLogic VNIC Configuration

The QLogic VNIC (Virtual Network Interface Card) driver in conjunction with the QLogic Ethernet Virtual I/O Controller (EVIC) provides virtual Ethernet interfaces and transport for Ethernet packets over Infiniband.

Users can modify NIC parameters through User Interface icon in Network Connections:
( Properties->"Configure..." button -> "Advanced" Tab).

Parameters available:

Vlan Id (802.1Q) 

  values from 0 to 4094 ( default 0, disabled )
  This specifies if VLAN ID-marked packet transmission is enabled and, if so, specifies the ID.

Priority (802.1P)

  values from 0 to 7 ( default 0, feature disabled)
  This specifies if priority-marked packet transmission is enabled.

Payload MTU size 

  values from 1500 to 9500 (default 1500)
  This specifies the maximum transfer unit size in 100 bytes increments.

Recv ChkSum offload 

  (default enabled)
  This specifies if IP protocols checksum calculations for receive is offloaded.

Send ChkSum offload

  (default enabled)
  This specifies if IP protocols checksum calculations for send is offloaded.
 

Secondary Path 

   (default disabled)
   Enabled - If more than one IB path to IOC exist then secondary IB instance of virtual port will be created and configured with the same parameters as primary one. Failover from Primary to Secondary IB path is transparent for user application sending data through associated NIC.

   Disabled – only one path at a time is allowed. If more than one path to IOC exists then failed path will be destroyed and next available path will be used for new connection. With this scenario there is a possibility new interface instance will be assigned different MAC address when other hosts compete for EVIC resources.
 

LBFO Bundle Id
   (default disabled) Enabling support for OS provided Load Balancing and Fail Over functionality on adapter level.
   If enabled group ID can be selected from predefined names.

 

Heartbeat interval

   configures interval for VNIC protocol heartbeat messages in milliseconds.
   0 – heartbeats disabled.

Note:
   To take advantage of the features supported by these options, ensure that the Ethernet gateway is also configured appropriately.  For example, if the Payload MTU for a VNIC interface is set to 4000, the MTU at the EVIC module must also be set at least 4000 for the setting to take effect.

<Return-to-Top>

 

DAT & uDAPL Configuration

In order for DAT/DAPL programs to execute correctly, the 'dat2.dll' file must be present in the current directory, %SystemRoot%\System32 or in the library search path.

OFED installation places the dat2.dll and dapl2.dll files in the '%SystemRoot%\system32' folder (a.k.a. \Windows).

The DAT/DAPL configuration file by convention is defined as '%SystemDrive%\DAT\dat.conf'. If the file '%SystemDrive%\DAT\dat.conf' does not exist, a template configuration file will be installed as '%SystemDrive%\DAT\dat.conf'.
In order to preserve existing installations, the dat.conf file is not automatically installed if '%SystemDrive%\DAT\dat.conf' exists.
A sample dat.conf file is always installed as '\Program Files\OFED\dat.conf '.

The default DAT configuration file specification can be overridden by use of the environment variable DAT_OVERRIDE. Such that 'DAT_OVERRIDE=D:\MyDAT\test.conf' would be a valid override specification.

Within the dat.conf file, the DAPL library specification can be located as the 5th whitespace separated line argument. By default the DAPL library file is installed as %WINDIR%\dapl2.dll'.

Should you choose to relocate the DAPL library file to a folder where whitespace appears in the full library path specification, then the full library file specification must be contained within double-quotes. A side effect of the double-quotes is the library specification is treated as a Windows string which implies the '\' (backslash character) is treated as an 'escape' character.  Hence all backslashes in the library path must be duplicated when enclosed in double-quotes (e.g., "C:\\Programs Files\\OFED\\dapl.dll").

uDAT/uDAPL version 2.0 runtime libraries are identified as dat2.dll and dapl2.dll, both in %SystemRoot%; see manual for further details..
 

<Return-to-Top>

 

SRP (SCSI RDMA Protocol) Driver Installation

The OFED installer does not install the SRP driver as part of a default installation.  If the SRP feature is selected in the custom installation window, an InfiniBand SRP Miniport driver will be installed; see the device manager view under SCSI and RAID controllers.

The system device 'InfiniBand I/O Unit' (IOU) device is required for correct SRP operation.  The OFED installer will install/load the IOU driver if the SRP feature is selected.  See the device manager view System Devices --> InfiniBand I/O Unit for conformation of correct IOU driver loading.

In order for the SRP miniport driver installation to complete, an SRP target must be detected by a Subnet Manager running somewhere on the InfiniBand fabric; either a local or remote Subnet Manager works.
When the subnet manager communicates the existence of an SRP target, Windows PNP (Plug-n-Play) will match the IOC device ID with DriverStore installed drives and then will load the IBiou.sys driver along with the ib_srp.sys driver.

An invocation of 'diskmgmt.msc' will display available SRP target disks.

Consult the OFED manual for further SRP topics including configuring a Linux OFED SRP target.

SRP Driver Uninstall

If the SRP (SCSI RDMA Protocol) driver has been previously installed, then in order to achieve a 'clean' uninstall, the SRP target drive(s) must be released (set offline in diskmgmt.msc).

The consequences of not releasing the SRP target drive(s) are that after the OFED uninstall reboot there are lingering InfiniBand driver files. These driver files remain because while the SRP target is active they have references, thus when the OFED uninstall attempts to delete the files the operation fails.


<Return-to-Top>

 

WinVerbs

WinVerbs is a userspace verbs and communication management interface optimized for the Windows operating system. Its lower interface is designed to support any RDMA based device, including Infiniband and iWarp. Its upper interface is capable of providing a low latency verbs interface, plus supports Microsoft's NetworkDirect Interface, DAPL, and OFED libibverbs interfaces. It consists of
a userspace library and a kernel filter driver.

The WinVerbs and WinMAD drivers load as upper filter drivers for the Infiniband HCA device.
(Open source iWarp drivers for Windows are not yet available.)  A corresponding
WinVerbs.dll and libibmad.dll userspace libraries install as part of the Winverbs driver installation package.
Additionally, a Windows port of the OFED libibverbs library, several test programs and OFED InfiniBand diagnostic utilities are also included.

As of OFED 2.1, Winverbs components are now integral components of a default OFED installation.
Although WinVerbs, WinMAD drivers, OFED libraries and utilities are install selectable features, they are automatically included as part of the default OFED install.

<Return-to-Top>
 

Known Issues as of 06/25/2013

<Return-to-Top>
 

RoCE Installation and Operation

RoCE (RDMA over Coherent/converged Ethernet) is only supported for Mellanox ConnectX-2/3 HCAs on 'x64' platforms.
Normally a RoCE HCAs are connected thru a RoCE 10gb Ethernet switch (DCA enabled), although port-to-port (back-to-back) RoCE connections are supported.

Since Mellanox declined to provide RoCE Ethernet driver sources (binary drivers only), the RoCE installation is outside the scope of the standard winOFED installation process.

First and foremost, install winOFED (3.1 and beyond).

Once winOFED is installed on your x64 platform disable the OpenSM service if it is running.
Now install RoCE Ethernet driver (msiexec /I RoCE_Ethernet_win7_x64.msi') or Right-click the .msi file.

The RoCE Ethernet driver 'mlx4eth62.sys' driver will be installed; from the Device Manager view (devmgmt.msc) 'Network adapters' view will show the 'OpenFabrics  IPoIB adapter' will be replaced with 'Mellanox ConnectX 10gb Ethernet Adapter'.

Once the RoCE Ethernet driver is installed, the installer will bounce (disable/enable) the Mellanox mlx4 bus driver (Mellanox ConnectX VPI, under device manager 'system devices').
The RoCE Ethernet installer notifier boxes will timeout or just click the 'OK" button.
If for some reason the installer VPI device bounce does not work, one can manually disable/enable the Mellanox VPI device.

After the RoCE Ethernet device installer has exited successfully, you need to assign an IP address to the network interface (ncpa.cpl, right-click the 'Mellanox 10gb Ethernet Local Area Connection' --> Properties --> Internet Protocol Version 4/6) and assign DHCP or manual IP address assignment.

RoCE Ethernet device removal can be accomplished via the 'Control Panel --> Programs and Features --> RoCE Ethernet Driver --> Uninstall'.

<Return-to-Top>