Open Fabrics Enterprise Distribution (OFED) Version 1.4 Installation Guide December 2008 ============================================================================== Table of contents ============================================================================== 1. Overview 2. Contents of the OFED Distribution 3. Hardware and Software Requirements 4. How to Download and Extract the OFED Distribution 5. Installing OFED Software 6. Building OFED RPMs 7. IPoIB Configuration 8. Uninstalling OFED 9. Upgrading OFED 10. Configuration 11. Related Documentation ============================================================================== 1. Overview ============================================================================== This is the OpenFabrics Enterprise Distribution (OFED) version 1.4 software package supporting InfiniBand and iWARP fabrics. It is composed of several software modules intended for use on a computer cluster constructed as an InfiniBand subnet or an iWARP network. This document describes how to install the various modules and test them in a Linux environment. General Notes: 1) The install script removes all previously installed OFED packages and re-installs from scratch. (Note: Configuration files will not be removed). You will be prompted to acknowledge the deletion of the old packages. 2) When installing OFED on an entire [homogeneous] cluster, a common strategy is to install the software on one of the cluster nodes (perhaps on a shared file system such as NFS). The resulting RPMs, created under OFED-1.4/RPMS directory, can then be installed on all nodes in the cluster using any cluster-aware tools (such as pdsh). ============================================================================== 2. OFED Package Contents ============================================================================== The OFED Distribution package generates RPMs for installing the following: o OpenFabrics core and ULPs: - HCA drivers (mthca, mlx4, ipath, ehca) - iWARP driver (cxgb3, nes) - core - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER Initiator and target, RDS, qlgc_vnic, uDAPL and NFS-RDMA o OpenFabrics utilities - OpenSM: InfiniBand Subnet Manager - Diagnostic tools - Performance tests o MPI - OSU MVAPICH stack supporting the InfiniBand and iWARP interface - Open MPI stack supporting the InfiniBand and iWARP interface - OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface - MPI benchmark tests (OSU BW/LAT, Intel MPI Benchmark, Presta) o Extra packages - open-iscsi: open-iscsi initiator with iSER support - ib-bonding: Bonding driver for IPoIB interface o Sources of all software modules (under conditions mentioned in the modules' LICENSE files) o Documentation ============================================================================== 3. Hardware and Software Requirements ============================================================================== 1) Server platform with InfiniBand HCA or iWARP RNIC (see OFED Distribution Release Notes for details) 2) Linux operating system (see OFED Distribution Release Notes for details) 3) Administrator privileges on your machine(s) 4) Disk Space: - For Build & Installation: 300MB - For Installation only: 200MB 5) For the OFED Distribution to compile on your machine, some software packages of your operating system (OS) distribution are required. These are listed here. OS Distribution Required Packages --------------- ---------------------------------- General: o Common to all gcc, glib, glib-devel, glibc, glibc-devel, glibc-devel-32bit (to build 32-bit libraries on x86_86 and ppc64), zlib-devel o RedHat, Fedora kernel-devel, rpm-build o SLES 10.0 kernel-source, rpm Note: To build 32-bit libraries on x86_64 and ppc64 platforms, the 32-bit glibc-devel should be installed. Specific Component Requirements: o Mvapich a Fortran Compiler (such as gcc-g77) o Mvapich2 libstdc++-devel, sysfsutils (SuSE), libsysfs-devel (RedHat5.0, Fedora C6) o Open MPI libstdc++-devel o ibutils tcl-8.4, tcl-devel-8.4, tk, libstdc++-devel o tvflash pciutils-devel o mstflint libstdc++-devel (32-bit on ppc64), gcc-c++ Note: The installer will warn you if you attempt to compile any of the above packages and do not have the prerequisites installed. *** Important Note for open-iscsi users: Installing iSER as part of OFED installation will also install open-iscsi. Before installing OFED, please uninstall any open-iscsi version that may be installed on your machine. Installing OFED with iSER support while another open-iscsi version is already installed will cause the installation process to fail. ============================================================================== 4. How to Download and Extract the OFED Distribution ============================================================================== 1) Download the OFED-X.X.X.tgz file to your target Linux host. If this package is to be installed on a cluster, it is recommended to download it to an NFS shared directory. 2) Extract the package using: tar xzvf OFED-X.X.X.tgz ============================================================================== 5. Installing OFED Software ============================================================================== 1) Go to the directory into which the package was extracted: cd /..../OFED-X.X.X 2) Installing the OFED package must be done as root. For a menu-driven first build and installation, run the installer script: ./install.pl Interactive menus will direct you through the install process. Note: After the installer completes, information about the OFED installation such as the prefix, the kernel version, and installation parameters can be found by running /etc/infiniband/info. Information on the driver version and source git trees can be found using the ofed_info utility During the interactive installation of OFED, two files are generated: ofed.conf and ofed_net.conf. ofed.conf holds the installed software modules and configuration settings chosen by the user. ofed_net.conf holds the IPoIB settings chosen by the user. If the package is installed on a cluster-shared directory, these files can then be used to perform an automatic, unattended installation of OFED on other machines in the cluster. The unattended installation will use the same choices as were selected in the interactive installation. For an automatic installation on any host, run the following: ./OFED-X.X.X/install.pl -c /ofed.conf -n /ofed_net.conf 3) Install script usage: Usage: ./install.pl [-c |--all|--hpc|--basic] [-n|--net ] -c|--config . Example of the config file can be found under docs (ofed.conf-example) -n|--net Example of the config file can be found under docs (ofed_net.conf-example) -l|--prefix Set installation prefix. -p|--print-available Print available packages for current platform. And create corresponding ofed.conf file. -k|--kernel . Default on this system: $(uname -r) -s|--kernel-sources . Default on this system: /lib/modules/$(uname -r)/build --build32 Build 32-bit libraries. Relevant for x86_64 and ppc64 platforms --without-depcheck Skip Distro's libraries check -v|-vv|-vvv. Set verbosity level -q. Set quiet - no messages will be printed --all|--hpc|--basic Install all,hpc or basic packages correspondingly Notes: ------ a. It is possible to rename and/or edit the ofed.conf and ofed_net.conf files. Thus it is possible to change user choices (observing the original format). See examples of ofed.conf and ofed_net.conf under OFED-X.X.X/docs. Run './install.pl -p' to get ofed.conf with all available packages included. b. Important note for open-iscsi users: Installing iSER as part of the OFED installation will also install open-iscsi. Before installing OFED, please uninstall any open-iscsi version that may be installed on your machine. Installing OFED with iSER support while another open-iscsi version is already installed will cause the installation process to fail. Install Process Results: ------------------------ o The OFED package is installed under directory. Default prefix is /usr o The kernel modules are installed under: - Infiniband subsystem: /lib/modules/`uname -r`/updates/kernel/drivers/infiniband/ - open-iscsi: /lib/modules/`uname -r`/updates/kernel/drivers/scsi/ - Chelsio driver: /lib/modules/`uname -r`/updates/kernel/drivers/net/cxgb3/cxgb3.ko - ConnectX driver: /lib/modules/`uname -r`/updates/kernel/drivers/net/mlx4/mlx4_core.ko - RDS: /lib/modules/`uname -r`/updates/kernel/net/rds/rds.ko - Bonding module: /lib/modules/`uname -r`/updates/kernel/drivers/net/bonding/bonding.ko o The package kernel include files are placed under /src/ofa_kernel/. These includes should be used when building kernel modules which use the Openfabrics stack. (Note that these includes, if needed, are "backported" to your kernel). o The raw package (un-backported) source files are placed under /src/ofa_kernel-1.4 o The script "openibd" is installed under /etc/init.d/. This script can be used to load and unload the software stack. o The directory /etc/infiniband is created with the files "info" and "openib.conf". The "info" script can be used to retrieve OFED installation information. The "openib.conf" file contains the list of modules that are loaded when the "openibd" script is used. o The file "90-ib.rules" is installed under /etc/udev/rules.d/ o If libibverbs-utils is installed, then ofed.sh and ofed.csh are installed under /etc/profile.d/. These automatically update the PATH environment variable with /bin. In addition, ofed.conf is installed under /etc/ld.so.conf.d/ to update the dynamic linker's run-time search path to find the InfiniBand shared libraries. o The file /etc/modprobe.conf is updated to include the following: - "alias ib ib_ipoib" for each ib interface. - "alias net-pf-27 ib_sdp" for sdp. o If opensm is installed, the daemon opensmd is installed under /etc/init.d/ o All verbs tests and examples are installed under /bin and management utilities under /sbin o ofed_info script provides information on the OFED version and git repository. o If IPoIB configuration files are included, ifcfg-ib files will be installed at: - RedHat: /etc/sysconfig/network-scripts/ - SuSE: /etc/sysconfig/network/ o If iSER is included, open-iscsi user-space files will be also installed: - Configuration files will be installed at /etc/iscsi - Startup script will be installed at: - RedHat: /etc/init.d/iscsi - SuSE: /etc/init.d/open-iscsi - Other tools (iscsiadm, iscsid, iscsi_discovery, iscsi-iname, iscsistart) will be installed under /sbin. - Documentation will be installed under: - RedHat: /usr/share/doc/iscsi-initiator-utils- - SuSE: /usr/share/doc/packages/open-iscsi o man pages will be installed under /usr/share/man/. ============================================================================== 6. Building OFED RPMs ============================================================================== 1) Go to the directory into which the package was extracted: cd /..../OFED-X.X.X 2) Run install.pl as explained above This script also builds OFED binary RPMs under OFED-X.X.X/RPMS; the sources are placed in OFED-X.X.X/SRPMS/. Once the install process has completed, the user may run ./install.pl on other machines that have the same operating system and kernel to install the new RPMs. Note: Depending on your hardware, the build procedure may take 30-45 minutes. Installation, however, is a relatively short process (~5 minutes). A common strategy for OFED installation on large homogeneous clusters is to extract the tarball on a network file system (such as NFS), build OFED RPMs on NFS, and then run the installer on each node with the RPMs that were previously built. ============================================================================== 7. IP-over-IB (IPoIB) Configuration ============================================================================== Configuring IPoIB is an optional step during the installation. During an interactive installation, the user may choose to insert the ifcfg-ib files. If this option is chosen, the ifcfg-ib files will be installed under: - RedHat: /etc/sysconfig/network-scripts/ - SuSE: /etc/sysconfig/network/ Setting IPoIB Configuration: ---------------------------- There is no default configuration for IPoIB interfaces. One should manually specify the full IP configuration during the interactive installation: IP address, network address, netmask, and broadcast address, or use the ofed_net.conf file. For bonding setting please see "ipoib_release_notes.txt" For unattended installations, a configuration file can be provided with this information. The configuration file must specify the following information: - Fixed values for each IPoIB interface - Base IPoIB configuration on Ethernet configuration (may be useful for cluster configuration) Here are some examples of ofed_net.conf: # Static settings; all values provided by this file IPADDR_ib0=172.16.0.4 NETMASK_ib0=255.255.0.0 NETWORK_ib0=172.16.0.0 BROADCAST_ib0=172.16.255.255 ONBOOT_ib0=1 # Based on eth0; each '*' will be replaced by the script with corresponding # octet from eth0. LAN_INTERFACE_ib0=eth0 IPADDR_ib0=172.16.'*'.'*' NETMASK_ib0=255.255.0.0 NETWORK_ib0=172.16.0.0 BROADCAST_ib0=172.16.255.255 ONBOOT_ib0=1 # Based on the first eth interface that is found (for n=0,1,...); # each '*' will be replaced by the script with corresponding octet from eth. LAN_INTERFACE_ib0= IPADDR_ib0=172.16.'*'.'*' NETMASK_ib0=255.255.0.0 NETWORK_ib0=172.16.0.0 BROADCAST_ib0=172.16.255.255 ONBOOT_ib0=1 ============================================================================== 8. Uninstalling OFED ============================================================================== There are two ways to uninstall OFED: 1) Via the installation menu. 2) Using the script ofed_uninstall.sh. The script is part of ofed-scripts package. 3) ofed_uninstall.sh script supports an option to executes 'openibd stop' before removing the RPMs using the flag: --unload-modules ============================================================================== 9. Upgrading OFED ============================================================================== If an old OFED version is installed, it may be upgraded by installing a new OFED version as described in section 5. Note that if the old OFED version was loaded before upgrading, you need to restart OFED or reboot your machine in order to start the new OFED stack. ============================================================================== 10. Configuration ============================================================================== Most of the OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files. The list of the modules that will be loaded automatically upon boot can be found in the /etc/infiniband/openib.conf file. Other configuration files include: - SDP configuration file: /etc/libsdp.conf - OpenSM configuration file: /etc/ofa/opensm.conf (for RedHat) /etc/sysconfig/opensm (for SuSE) - should be created manually if required. - DAPL configuration file: /etc/dat.conf See packages Release Notes for more details. Note: After the installer completes, information about the OFED installation such as the prefix, kernel version, and installation parameters can be found by running /etc/infiniband/info. ============================================================================== 11. Related Documentation ============================================================================== OFED documentation is located in the ofed-docs RPM. After installation the documents are located under the directory: /usr/share/doc/ofed-docs-1.4 for RedHat /usr/share/doc/packages/ofed-docs-1.4 for SuSE Document list: o README.txt o OFED_Installation_Guide.txt o MPI_README.txt o Examples of configuration files o OFED_tips.txt o HOWTO.build_ofed o All release notes and README files For more information, please visit the OpenFabrics web site: http://www.openfabrics.org/ open-iscsi documentation is located at: - RedHat: /usr/share/doc/iscsi-initiator-utils- - SuSE: /usr/share/doc/packages/open-iscsi For more information, please visit the open-iscsi web site: http://www.open-iscsi.org/