久久精品国产精品国产精品污,男人扒开添女人下部免费视频,一级国产69式性姿势免费视频,夜鲁夜鲁很鲁在线视频 视频,欧美丰满少妇一区二区三区,国产偷国产偷亚洲高清人乐享,中文 在线 日韩 亚洲 欧美,熟妇人妻无乱码中文字幕真矢织江,一区二区三区人妻制服国产

歡迎訪問 生活随笔!

生活随笔

當(dāng)前位置: 首頁 > 运维知识 > linux >内容正文

linux

linux安装软件命令1003无标题,linux系统安装OFED(infiniband)

發(fā)布時(shí)間:2025/4/5 linux 34 豆豆
生活随笔 收集整理的這篇文章主要介紹了 linux安装软件命令1003无标题,linux系统安装OFED(infiniband) 小編覺得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.

計(jì)算節(jié)點(diǎn)安裝OFEDnode

scientific linux5.8 x86_64linux

1.查看系統(tǒng)系統(tǒng):c++

[root@localhost ~]# cat /etc/redhat-release:git

Scientific Linux release 5.8 (Boron)api

[root@localhost ~]# uname -a網(wǎng)絡(luò)

Linux localhost.localdomain 2.6.18-308.1.1.el5 #1 SMP Thu Mar 8 09:54:23 EST 2012 x86_64 x86_64 x86_64 GNU/Linuxdom

2.查看分區(qū)信息ide

[root@localhost ~]# df -h工具

FilesystemSizeUsed Avail Use% Mounted onpost

/dev/sda61.5T776M1.5T1% /

/dev/sda59.7G2.7G6.6G29% /usr

/dev/sda339G177M37G1% /home

/dev/sda1487M18M444M4% /boot

tmpfs16G016G0% /dev/shm

/dev/sr14.1G4.1G0 100% /media

3.infinband硬件信息:Mellanox

[root@localhost ~]# lspci |grep Mellanox

04:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

4.解決依賴包:掛在使用本地yum

1.備份分并刪除/etc/yum.repo.d/目錄下的yum配置文件,

2.新建一個(gè)本地源配置文件123.repo,文件內(nèi)容以下

[root@localhost ~]# cat /etc/yum.repos.d/123.repo

[dvdserver]

name=yumserver

baseurl=file:///media/SL

enabled=1

gpgcheck=0

3固然你必須使用mount掛在光盤到/media

4.解決依賴包

(1)gcc-c++

yum -y install gcc-c++

Loaded plugins: kernel-module

dvdserver| 1.3 kB00:00

dvdserver/primary| 1.0 MB 00:00

dvdserver3725/3725

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package gcc-c++.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: libstdc++-devel = 4.1.2-52.el5 for package: gcc-c++

--> Processing Dependency: gcc = 4.1.2-52.el5 for package: gcc-c++

--> Running transaction check

---> Package gcc.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc

---> Package libstdc++-devel.x86_64 0:4.1.2-52.el5 set to be updated

dvdserver/filelists| 3.4 MB00:01

--> Running transaction check

---> Package glibc-devel.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: glibc-headers = 2.5-81.el5_8.1 for package: glibc-devel

--> Processing Dependency: glibc-headers for package: glibc-devel

--> Running transaction check

---> Package glibc-headers.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers

--> Processing Dependency: kernel-headers for package: glibc-headers

--> Running transaction check

---> Package kernel-headers.x86_64 0:2.6.18-308.1.1.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

gcc-c++x86_644.1.2-52.el5dvdserver3.8 M

Installing for dependencies:

gccx86_644.1.2-52.el5dvdserver5.3 M

glibc-develx86_642.5-81.el5_8.1dvdserver2.4 M

glibc-headersx86_642.5-81.el5_8.1dvdserver597 k

kernel-headersx86_642.6.18-308.1.1.el5dvdserver1.4 M

libstdc++-develx86_644.1.2-52.el5dvdserver2.8 M

Transaction Summary

================================================================================

Install6 Package(s)

Upgrade0 Package(s)

Total download size: 16 M

Downloading Packages:

--------------------------------------------------------------------------------

Total1.5 GB/s |16 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-headers1/6

Installing: glibc-headers2/6

Installing: glibc-devel3/6

Installing: libstdc++-devel4/6

Installing: gcc5/6

Installing: gcc-c++6/6

Installed:

gcc-c++.x86_64 0:4.1.2-52.el5

Dependency Installed:

gcc.x86_64 0:4.1.2-52.el5

glibc-devel.x86_64 0:2.5-81.el5_8.1

glibc-headers.x86_64 0:2.5-81.el5_8.1

kernel-headers.x86_64 0:2.6.18-308.1.1.el5

libstdc++-devel.x86_64 0:4.1.2-52.el5

Complete!

(2)glib

[root@localhost ~]# yum -y install glib

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib.i386 1:1.2.10-20.el5 set to be updated

---> Package glib.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glibi3861:1.2.10-20.el5dvdserver137 k

glibx86_641:1.2.10-20.el5dvdserver144 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 281 k

Downloading Packages:

--------------------------------------------------------------------------------

Total159 MB/s | 281 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib1/2

Installing: glib2/2

Installed:

glib.i386 1:1.2.10-20.el5glib.x86_64 1:1.2.10-20.el5

Complete!

(3) glib-devel

[root@localhost ~]# yum -y install glib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib-devel.i386 1:1.2.10-20.el5 set to be updated

---> Package glib-devel.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glib-develi3861:1.2.10-20.el5dvdserver39 k

glib-develx86_641:1.2.10-20.el5dvdserver39 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 77 k

Downloading Packages:

--------------------------------------------------------------------------------

Total44 MB/s |77 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib-devel1/2

Installing: glib-devel2/2

Installed:

glib-devel.i386 1:1.2.10-20.el5glib-devel.x86_64 1:1.2.10-20.el5

Complete!

[root@localhost ~]# yum -y install kernel-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package kernel-devel.x86_64 0:2.6.18-308.1.1.el5 set to be installed

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

kernel-develx86_642.6.18-308.1.1.el5dvdserver5.7 M

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 5.7 M

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-devel1/1

Installed:

kernel-devel.x86_64 0:2.6.18-308.1.1.el5

Complete!

(4)zilb-devel

[root@localhost ~]# yum -y install zlib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package zlib-devel.i386 0:1.2.3-4.el5 set to be updated

---> Package zlib-devel.x86_64 0:1.2.3-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

zlib-develi3861.2.3-4.el5dvdserver101 k

zlib-develx86_641.2.3-4.el5dvdserver103 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 204 k

Downloading Packages:

--------------------------------------------------------------------------------

Total743 kB/s | 204 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: zlib-devel1/2

Installing: zlib-devel2/2

Installed:

zlib-devel.i386 0:1.2.3-4.el5zlib-devel.x86_64 0:1.2.3-4.el5

Complete!

(5)rpm-build

[root@localhost ~]# yum -y install rpm-build

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package rpm-build.x86_64 0:4.4.2.3-28.el5_8 set to be updated

--> Processing Dependency: xz for package: rpm-build

--> Processing Dependency: elfutils for package: rpm-build

--> Running transaction check

---> Package elfutils.x86_64 0:0.137-3.el5 set to be updated

--> Processing Dependency: elfutils-libs-x86_64 = 0.137-3.el5 for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.130)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.127)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.126)(64bit) for package: elfutils

--> Processing Dependency: libasm.so.1(ELFUTILS_1.0)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.122)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1()(64bit) for package: elfutils

--> Processing Dependency: libasm.so.1()(64bit) for package: elfutils

---> Package xz.x86_64 0:4.999.9-0.3.beta.20091007git.el5 set to be updated

--> Processing Dependency: xz-libs = 4.999.9-0.3.beta.20091007git.el5 for package: xz

--> Processing Dependency: liblzma.so.0()(64bit) for package: xz

--> Running transaction check

---> Package elfutils-libs.x86_64 0:0.137-3.el5 set to be updated

---> Package xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

rpm-buildx86_644.4.2.3-28.el5_8dvdserver303 k

Installing for dependencies:

elfutilsx86_640.137-3.el5dvdserver216 k

elfutils-libsx86_640.137-3.el5dvdserver182 k

xzx86_644.999.9-0.3.beta.20091007git.el5dvdserver146 k

xz-libsx86_644.999.9-0.3.beta.20091007git.el5dvdserver94 k

Transaction Summary

================================================================================

Install5 Package(s)

Upgrade0 Package(s)

Total download size: 941 k

Downloading Packages:

--------------------------------------------------------------------------------

Total189 MB/s | 941 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: xz-libs1/5

Installing: xz2/5

Installing: elfutils-libs3/5

Installing: elfutils4/5

Installing: rpm-build5/5

Installed:

rpm-build.x86_64 0:4.4.2.3-28.el5_8

Dependency Installed:

elfutils.x86_64 0:0.137-3.el5

elfutils-libs.x86_64 0:0.137-3.el5

xz.x86_64 0:4.999.9-0.3.beta.20091007git.el5

xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el5

Complete!

(6)redhat-rpm-config

[root@localhost ~]# yum -y install redhat-rpm-config

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package redhat-rpm-config.noarch 0:8.0.45-32.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

redhat-rpm-confignoarch8.0.45-32.el5dvdserver55 k

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 55 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: redhat-rpm-config1/1

Installed:

redhat-rpm-config.noarch 0:8.0.45-32.el5

Complete!

(7)tcl

[root@localhost ~]# yum -y install tcl

Loaded plugins: kernel-module

Setting up Install Process

Package tcl-8.4.13-4.el5.x86_64 already installed and latest version

Resolving Dependencies

--> Running transaction check

---> Package tcl.i386 0:8.4.13-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

tcli3868.4.13-4.el5dvdserver956 k

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 956 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: tcl1/1

Installed:

tcl.i386 0:8.4.13-4.el5

Complete!

(8)tcl-devel

[root@localhost ~]# yum -y install tcl-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package tcl-devel.i386 0:8.4.13-4.el5 set to be updated

---> Package tcl-devel.x86_64 0:8.4.13-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

tcl-develi3868.4.13-4.el5dvdserver1.0 M

tcl-develx86_648.4.13-4.el5dvdserver1.0 M

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 2.0 M

Downloading Packages:

--------------------------------------------------------------------------------

Total786 MB/s | 2.0 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: tcl-devel1/2

Installing: tcl-devel2/2

Installed:

tcl-devel.i386 0:8.4.13-4.el5tcl-devel.x86_64 0:8.4.13-4.el5

Complete!

(9)libtool

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# yum -y install libtool

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package libtool.x86_64 0:1.5.22-7.el5_4 set to be updated

--> Processing Dependency: autoconf >= 2.50 for package: libtool

--> Processing Dependency: automake >= 1.4 for package: libtool

--> Running transaction check

---> Package autoconf.noarch 0:2.59-12 set to be updated

--> Processing Dependency: imake for package: autoconf

---> Package automake.noarch 0:1.9.6-2.3.el5 set to be updated

--> Running transaction check

---> Package imake.x86_64 0:1.0.2-3 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

libtoolx86_641.5.22-7.el5_4dvdserver667 k

Installing for dependencies:

autoconfnoarch2.59-12dvdserver648 k

automakenoarch1.9.6-2.3.el5dvdserver476 k

imakex86_641.0.2-3dvdserver318 k

Transaction Summary

================================================================================

Install4 Package(s)

Upgrade0 Package(s)

Total download size: 2.1 M

Downloading Packages:

--------------------------------------------------------------------------------

Total681 MB/s | 2.1 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: imake1/4

Installing: autoconf2/4

Installing: automake3/4

Installing: libtool4/4

Installed:

libtool.x86_64 0:1.5.22-7.el5_4

Dependency Installed:

autoconf.noarch 0:2.59-12automake.noarch 0:1.9.6-2.3.el5

imake.x86_64 0:1.0.2-3

Complete!

注意:計(jì)算節(jié)點(diǎn)到此依賴包安裝完成.注意安裝前必定要安裝上訴依賴包,不然你等著報(bào)錯(cuò)包

5.安裝驅(qū)動(dòng)

到mellanox官網(wǎng)下載對(duì)應(yīng)操做系統(tǒng)的版本,本案例因?yàn)閙ellanox官網(wǎng)沒有scientific linux5.8的相應(yīng)驅(qū)動(dòng),只能下載red hat5.8的操做系統(tǒng)使用源代碼當(dāng)前操做系統(tǒng)編譯安裝

MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.tgz

值得注意的是這個(gè)驅(qū)動(dòng)就算在Red Hat 5.8上安裝最好系統(tǒng)是英文版,最好/usr目錄10GB

(1)解壓到指定目錄,進(jìn)入/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# tar -xzvf MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.tgz -C /usr/src/

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# cd /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

目錄文件以下

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# ls

docsmlnx_add_kernel_support.shrepodatasrc

firmwaremlnxofedinstallRPMSuninstall.sh

(2)標(biāo)準(zhǔn)安裝就報(bào)錯(cuò)了,以下

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# ./mlnxofedinstall The 2.6.18-308.1.1.el5 kernel is installed, MLNX_OFED does not have drivers available for this kernel.

You can run mlnx_add_kernel_support.sh in order to to generate an MLNX_OFED package with drivers for this kernel.

根據(jù)提示添加kernel吧,固然也報(bào)錯(cuò),不詳細(xì)說了,沒辦法只有源代碼了以下:進(jìn)入源代碼src/

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# cd /usr/src/MLNX_O

FED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/

解壓源代碼包src/MLNX_OFED_SRC-1.5.3-4.0.42.tgz

解決方法:

(3).解壓出來的源代碼目錄以下,并執(zhí)行源代碼安裝了

[root@localhost src]# cd MLNX_OFED_SRC-1.5.3-4.0.42

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ls

BUILD_IDinstall.plLICENSEofed.confRPMSSRPMSuninstall.sh

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ./install.pl執(zhí)行腳本

OFED Distribution Software Installation Menu

1) View OFED Installation Guide

2) Install OFED Software(安裝OFED驅(qū)動(dòng))

3) Show Installed Software

4) Configure IPoIB(配置Ib網(wǎng)卡)

5) Uninstall OFED Software

Q) Exit

Select Option [1-5]:2

OFED Distribution Software Installation Menu

1) Basic (OFED modules and basic user level libraries)(只是驅(qū)動(dòng))

2) HPC (OFED modules and libraries, MPI and diagnostic tools)(驅(qū)動(dòng)和MPI工具)

3) All packages (all of Basic, HPC)(管理節(jié)點(diǎn)須要opensmd服務(wù),選擇個(gè)就對(duì)了)

4) Customize(自定義,若是你對(duì)這個(gè)infinband網(wǎng)絡(luò)結(jié)構(gòu)夠了解選它吧)

5) Packages required by VMA (IB and Eth)

6) Packages required by VMA IB

7) Packages required by VMA Eth

Q) Exit

Select Option [1-7]:1

Below is the list of OFED packages that you have chosen

(some may have been added by the installer due to package dependencies):

ofed-scripts

libibverbs

libibverbs-utils

libibverbs-devel

libmthca

libmverbs

libmverbs-devel

libmlx4

libcxgb3

libnes

libipathverbs

librdmacm

librdmacm-utils

mstflint

ofed-docs

mlnx-ofa_kernel

kernel-mft

過程以下

Build ofed-scripts RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.src.rpm

Install ofed-scripts RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

Build libibverbs RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libibverbs-1.1.5mlnx2-1.src.rpm

Install libibverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-utils-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-devel-1.1.5mlnx2-1.x86_64.rpm

Build libmthca RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libmthca-1.0.6mlnx1-0.1.gbe5eef3.src.rpm

Install libmthca RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmthca-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

Build libmverbs RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libmverbs-0.1.0-3.15.gd28970e.src.rpm

Install libmverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmverbs-0.1.0-3.15.gd28970e.x86_64.rpm

Install libmverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmverbs-devel-0.1.0-3.15.gd28970e.x86_64.rpm

Build libmlx4 RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libmlx4-1.0.2mlnx6-1.src.rpm

Install libmlx4 RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmlx4-1.0.2mlnx6-1.x86_64.rpm

Build libcxgb3 RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libcxgb3-1.3.1-1.src.rpm

Install libcxgb3 RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libcxgb3-1.3.1-1.x86_64.rpm

Build libnes RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libnes-1.1.1mlnx1-1.src.rpm

Install libnes RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libnes-1.1.1mlnx1-1.x86_64.rpm

Build libipathverbs RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libipathverbs-1.2mlnx1-1.src.rpm

Install libipathverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libipathverbs-1.2mlnx1-1.x86_64.rpm

Build librdmacm RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/librdmacm-1.0.15-1.src.rpm

Install librdmacm RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-1.0.15-1.x86_64.rpm

Install librdmacm-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-utils-1.0.15-1.x86_64.rpm

Build mstflint RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/mstflint-1.4mlnx4-1.21.gd948ddd.src.rpm

Install mstflint RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mstflint-1.4mlnx4-1.21.gd948ddd.x86_64.rpm

Build ofed-docs RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ofed-docs-1.5.3-OFED.1.5.3.4.0.42.src.rpm

Install ofed-docs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ofed-docs-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

Build mlnx-ofa_kernel RPM

Running rpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --nodeps --define '_dist .unsupported' --define 'configure_options--with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_ib-mod --with-mlx4_vnic-mod --with-nes-mod --with-qib-mod --with-ipoib-mod' --define 'build_kernel_ib 1' --define 'build_kernel_ib_devel 1' --define 'KVERSION 2.6.18-308.1.1.el5' --define 'K_SRC /lib/modules/2.6.18-308.1.1.el5/build' --define 'network_dir /etc/sysconfig/network-scripts' --define '_prefix /usr' --define '__arch_install_post %{nil}' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.src.rpm

Install mlnx-ofa_kernel RPM:

Installing mlnx-ofa_kernel RPM

cmd: rpm -ivh--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.unsupported.x86_64.rpm

Preparing...##################################################

mlnx-ofa_kernel##################################################

Installing kmod-mlnx-ofa_kernel RPM

Preparing...##################################################

kmod-mlnx-ofa_kernel##################################################

Build kernel-mft RPM

Running rpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'KVERSION 2.6.18-308.1.1.el5' --define 'K_SRC /lib/modules/2.6.18-308.1.1.el5/build' --define '_release 2.6.18_308.1.1.el5' --define 'debug_package %{nil}' --define '_prefix /usr' --define '__arch_install_post %{nil}' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/kernel-mft-2.7.1-7.src.rpm

Install kernel-mft RPM:

Running rpm -iv--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/kernel-mft-2.7.1-2.6.18_308.1.1.el5.x86_64.rpm

The default IPoIB interface configuration is based on DHCP.

Note that a special patch for DHCP is required for supporting IPoIB.

The patch is available under docs/dhcp

If you do not have DHCP, you must change this configuration in the following steps.

Do you want to configure ib0? [Y/n]:y輸入y配置你的ib卡文件

The current IPoIB configuration for ib0 is:

# Mellanox Technologies MT27500 Family [ConnectX-3]

DEVICE=ib0

BOOTPROTO=dhcp

HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:F4:52:14:03:00:0D:C1:A1

ONBOOT=no

HOTPLUG=no

DHCP_HOSTNAME=localhost.localdomain

Do you want to change this configuration? [y/N]:y須要修改嗎,固然y

Enter an IP Adress: 192.168.1.101(IP)

Enter the Netmask: 255.255.255.0(掩碼)

Enter the Network: 192.168.1.0(網(wǎng)絡(luò))

Enter the Broadcast Adress: 192.168.1.255(廣播地址)

Start Device On Boot? [Y/n]:y開機(jī)啟動(dòng)(依賴openibd服務(wù))

Selected configuration:

DEVICE=ib0

IPADDR=192.168.1.101

NETMASK=255.255.255.0

NETWORK=192.168.1.0

BROADCAST=192.168.1.255

ONBOOT=yes

Do you want to save the selected configuration? [Y/n]:y保存ib配置

Do you want to configure ib1? [Y/n]:n根據(jù)你需求了,我沒開

IPoIB interfaces configured successfully

Press any key to continue ...

Device (15b3:1003):

03:00.0 Network controller [0207]: Mellanox Technologies MT27500 Family [ConnectX-3]

Link Width: 8x

PCI Link Speed: Unknown

Installation finished successfully.

Press any key to continue...

OFED Distribution Software Installation Menu

1) View OFED Installation Guide

2) Install OFED Software

3) Show Installed Software

4) Configure IPoIB

5) Uninstall OFED Software

Q) Exit

Select Option [1-5]:q退出

Exiting

驅(qū)動(dòng)基本安裝完成了,哈哈完美........

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# chkconfig --level 3 openibd on

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# chkconfig --level 5 openibd on設(shè)置開啟啟動(dòng)服務(wù)

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# /etc/init.d/openibd restart手動(dòng)啟動(dòng)服務(wù)命令

Unloading HCA driver:[OK]

Loading HCA driver and Access Layer:[OK]

Setting up InfiniBand network interfaces:

Bringing up interface ib0:[OK]

Determining IP information for ib1... failed.

Bringing up interface ib1:[OK]

Setting up service network . . .[done]

查看ib狀態(tài)卡信息

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 1

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... OFED-internal-1.5.3-4.0.42: 2.6.18-308.1.1.el5

Host Driver RPM Check .................. PASS

Firmware on CA #0 VPI .................. v2.11.550

Firmware Check on CA #0 (VPI) .......... NA

REASON: NO required fw version

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 1

Port State of Port #1 on CA #0 (VPI)..... UP 4X FDR10 (InfiniBand)看到這個(gè)就OK了,是否UP取決與infiniband網(wǎng)絡(luò)中必需要有一個(gè)安裝并開啟opensmd服務(wù)

Port State of Port #2 on CA #0 (VPI)..... DOWN (InfiniBand)

Error Counter Check on CA #0 (VPI)...... PASS

Kernel Syslog Check .................... PASS

Node GUID on CA #0 (VPI) ............... f4:52:14:03:00:0d:c1:a0

------------------ DONE ---------------------

能夠測(cè)試ping

rtt min/avg/max/mdev = 0.144/0.521/0.899/0.378 ms

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ping 192.168.1.201

PING 192.168.1.201 (192.168.1.201) 56(84) bytes of data.

64 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=3.69 ms

64 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.104 ms

64 bytes from 192.168.1.201: icmp_seq=3 ttl=64 time=0.119 ms

--- 192.168.1.201 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2001ms

rtt min/avg/max/mdev = 0.104/1.305/3.693/1.688 ms

算Ok了啦,值得注意的是若是infiniband網(wǎng)絡(luò)中必需要有一個(gè)安裝并開啟opensmd服務(wù),否者你infiniband卡狀態(tài)永遠(yuǎn)down。

那咱們看看管理節(jié)點(diǎn)安裝了

scientific linux5.8 x86_64

1.查看系統(tǒng)系統(tǒng):

[root@localhost ~]# cat /etc/redhat-release:

Scientific Linux release 5.8 (Boron)

[root@localhost ~]# uname -a

Linux localhost.localdomain 2.6.18-308.1.1.el5 #1 SMP Thu Mar 8 09:54:23 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

2.查看分區(qū)信息

[root@localhost ~]# df -h

FilesystemSizeUsed Avail Use% Mounted on

/dev/sda61.5T776M1.5T1% /

/dev/sda59.7G2.7G6.6G29% /usr

/dev/sda339G177M37G1% /home

/dev/sda1487M18M444M4% /boot

tmpfs16G016G0% /dev/shm

/dev/sr14.1G4.1G0 100% /media

3.infinband硬件信息:Mellanox

[root@localhost ~]# lspci |grep Mellanox

04:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

4.解決依賴包:掛在使用本地yum

1.備份分并刪除/etc/yum.repo.d/目錄下的yum配置文件,

2.新建一個(gè)本地源配置文件123.repo,文件內(nèi)容以下

[root@localhost ~]# cat /etc/yum.repos.d/123.repo

[dvdserver]

name=yumserver

baseurl=file:///media/SL

enabled=1

gpgcheck=0

3固然你必須使用mount掛在光盤到/media

4.解決依賴包

(1)gcc-c++

yum -y install gcc-c++

Loaded plugins: kernel-module

dvdserver| 1.3 kB00:00

dvdserver/primary| 1.0 MB00:00

dvdserver3725/3725

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package gcc-c++.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: libstdc++-devel = 4.1.2-52.el5 for package: gcc-c++

--> Processing Dependency: gcc = 4.1.2-52.el5 for package: gcc-c++

--> Running transaction check

---> Package gcc.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc

---> Package libstdc++-devel.x86_64 0:4.1.2-52.el5 set to be updated

dvdserver/filelists| 3.4 MB00:01

--> Running transaction check

---> Package glibc-devel.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: glibc-headers = 2.5-81.el5_8.1 for package: glibc-devel

--> Processing Dependency: glibc-headers for package: glibc-devel

--> Running transaction check

---> Package glibc-headers.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers

--> Processing Dependency: kernel-headers for package: glibc-headers

--> Running transaction check

---> Package kernel-headers.x86_64 0:2.6.18-308.1.1.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

gcc-c++x86_644.1.2-52.el5dvdserver3.8 M

Installing for dependencies:

gccx86_644.1.2-52.el5dvdserver5.3 M

glibc-develx86_642.5-81.el5_8.1dvdserver2.4 M

glibc-headersx86_642.5-81.el5_8.1dvdserver597 k

kernel-headersx86_642.6.18-308.1.1.el5dvdserver1.4 M

libstdc++-develx86_644.1.2-52.el5dvdserver2.8 M

Transaction Summary

================================================================================

Install6 Package(s)

Upgrade0 Package(s)

Total download size: 16 M

Downloading Packages:

--------------------------------------------------------------------------------

Total1.5 GB/s |16 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-headers1/6

Installing: glibc-headers2/6

Installing: glibc-devel3/6

Installing: libstdc++-devel4/6

Installing: gcc5/6

Installing: gcc-c++6/6

Installed:

gcc-c++.x86_64 0:4.1.2-52.el5

Dependency Installed:

gcc.x86_64 0:4.1.2-52.el5

glibc-devel.x86_64 0:2.5-81.el5_8.1

glibc-headers.x86_64 0:2.5-81.el5_8.1

kernel-headers.x86_64 0:2.6.18-308.1.1.el5

libstdc++-devel.x86_64 0:4.1.2-52.el5

Complete!

(2)glib

[root@localhost ~]# yum -y install glib

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib.i386 1:1.2.10-20.el5 set to be updated

---> Package glib.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glibi3861:1.2.10-20.el5 dvdserver137 k

glibx86_641:1.2.10-20.el5dvdserver144 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 281 k

Downloading Packages:

--------------------------------------------------------------------------------

Total159 MB/s | 281 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib1/2

Installing: glib2/2

Installed:

glib.i386 1:1.2.10-20.el5glib.x86_64 1:1.2.10-20.el5

Complete!

(3) glib-devel

[root@localhost ~]# yum -y install glib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib-devel.i386 1:1.2.10-20.el5 set to be updated

---> Package glib-devel.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glib-develi3861:1.2.10-20.el5dvdserver39 k

glib-develx86_641:1.2.10-20.el5dvdserver39 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 77 k

Downloading Packages:

--------------------------------------------------------------------------------

Total44 MB/s |77 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib-devel1/2

Installing: glib-devel2/2

Installed:

glib-devel.i386 1:1.2.10-20.el5glib-devel.x86_64 1:1.2.10-20.el5

Complete!

[root@localhost ~]# yum -y install kernel-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package kernel-devel.x86_64 0:2.6.18-308.1.1.el5 set to be installed

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

kernel-develx86_642.6.18-308.1.1.el5dvdserver5.7 M

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 5.7 M

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-devel1/1

Installed:

kernel-devel.x86_64 0:2.6.18-308.1.1.el5

Complete!

(4)zilb-devel

[root@localhost ~]# yum -y install zlib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package zlib-devel.i386 0:1.2.3-4.el5 set to be updated

---> Package zlib-devel.x86_64 0:1.2.3-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

zlib-develi3861.2.3-4.el5dvdserver101 k

zlib-develx86_641.2.3-4.el5dvdserver103 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 204 k

Downloading Packages:

--------------------------------------------------------------------------------

Total743 kB/s | 204 kB 00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: zlib-devel1/2

Installing: zlib-devel2/2

Installed:

zlib-devel.i386 0:1.2.3-4.el5zlib-devel.x86_64 0:1.2.3-4.el5

Complete!

(5)rpm-build

[root@localhost ~]# yum -y install rpm-build

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package rpm-build.x86_64 0:4.4.2.3-28.el5_8 set to be updated

--> Processing Dependency: xz for package: rpm-build

--> Processing Dependency: elfutils for package: rpm-build

--> Running transaction check

---> Package elfutils.x86_64 0:0.137-3.el5 set to be updated

--> Processing Dependency: elfutils-libs-x86_64 = 0.137-3.el5 for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.130)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.127)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.126)(64bit) for package: elfutils

--> Processing Dependency: libasm.so.1(ELFUTILS_1.0)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.122)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1()(64bit) for package: elfutils

--> Processing Dependency: libasm.so.1()(64bit) for package: elfutils

---> Package xz.x86_64 0:4.999.9-0.3.beta.20091007git.el5 set to be updated

--> Processing Dependency: xz-libs = 4.999.9-0.3.beta.20091007git.el5 for package: xz

--> Processing Dependency: liblzma.so.0()(64bit) for package: xz

--> Running transaction check

---> Package elfutils-libs.x86_64 0:0.137-3.el5 set to be updated

---> Package xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

rpm-buildx86_644.4.2.3-28.el5_8dvdserver303 k

Installing for dependencies:

elfutilsx86_640.137-3.el5dvdserver216 k

elfutils-libsx86_640.137-3.el5dvdserver182 k

xzx86_644.999.9-0.3.beta.20091007git.el5dvdserver146 k

xz-libsx86_644.999.9-0.3.beta.20091007git.el5dvdserver94 k

Transaction Summary

================================================================================

Install5 Package(s)

Upgrade0 Package(s)

Total download size: 941 k

Downloading Packages:

--------------------------------------------------------------------------------

Total189 MB/s | 941 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: xz-libs1/5

Installing: xz2/5

Installing: elfutils-libs3/5

Installing: elfutils4/5

Installing: rpm-build5/5

Installed:

rpm-build.x86_64 0:4.4.2.3-28.el5_8

Dependency Installed:

elfutils.x86_64 0:0.137-3.el5

elfutils-libs.x86_64 0:0.137-3.el5

xz.x86_64 0:4.999.9-0.3.beta.20091007git.el5

xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el5

Complete!

(6)redhat-rpm-config

[root@localhost ~]# yum -y install redhat-rpm-config

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package redhat-rpm-config.noarch 0:8.0.45-32.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

redhat-rpm-confignoarch8.0.45-32.el5dvdserver55 k

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 55 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: redhat-rpm-config1/1

Installed:

redhat-rpm-config.noarch 0:8.0.45-32.el5

Complete!

(7)tcl

[root@localhost ~]# yum -y install tcl

Loaded plugins: kernel-module

Setting up Install Process

Package tcl-8.4.13-4.el5.x86_64 already installed and latest version

Resolving Dependencies

--> Running transaction check

---> Package tcl.i386 0:8.4.13-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

tcli3868.4.13-4.el5dvdserver956 k

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 956 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: tcl1/1

Installed:

tcl.i386 0:8.4.13-4.el5

Complete!

(8)tcl-devel

[root@localhost ~]# yum -y install tcl-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package tcl-devel.i386 0:8.4.13-4.el5 set to be updated

---> Package tcl-devel.x86_64 0:8.4.13-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

tcl-develi3868.4.13-4.el5dvdserver1.0 M

tcl-develx86_648.4.13-4.el5dvdserver1.0 M

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 2.0 M

Downloading Packages:

--------------------------------------------------------------------------------

Total786 MB/s | 2.0 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: tcl-devel1/2

Installing: tcl-devel2/2

Installed:

tcl-devel.i386 0:8.4.13-4.el5tcl-devel.x86_64 0:8.4.13-4.el5

Complete!

(9)libtool

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# yum -y install libtool

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package libtool.x86_64 0:1.5.22-7.el5_4 set to be updated

--> Processing Dependency: autoconf >= 2.50 for package: libtool

--> Processing Dependency: automake >= 1.4 for package: libtool

--> Running transaction check

---> Package autoconf.noarch 0:2.59-12 set to be updated

--> Processing Dependency: imake for package: autoconf

---> Package automake.noarch 0:1.9.6-2.3.el5 set to be updated

--> Running transaction check

---> Package imake.x86_64 0:1.0.2-3 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

libtoolx86_641.5.22-7.el5_4dvdserver667 k

Installing for dependencies:

autoconfnoarch2.59-12dvdserver648 k

automakenoarch1.9.6-2.3.el5dvdserver476 k

imakex86_641.0.2-3dvdserver318 k

Transaction Summary

================================================================================

Install4 Package(s)

Upgrade0 Package(s)

Total download size: 2.1 M

Downloading Packages:

--------------------------------------------------------------------------------

Total681 MB/s | 2.1 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: imake1/4

Installing: autoconf2/4

Installing: automake3/4

Installing: libtool4/4

Installed:

libtool.x86_64 0:1.5.22-7.el5_4

Dependency Installed:

autoconf.noarch 0:2.59-12automake.noarch 0:1.9.6-2.3.el5

imake.x86_64 0:1.0.2-3

Complete!

到此依賴包安裝完成.注意安裝前必定要安裝上訴依賴包,不然你等著報(bào)錯(cuò)包

5.安裝驅(qū)動(dòng)

到mellanox官網(wǎng)下載對(duì)應(yīng)操做系統(tǒng)的版本,本案例因?yàn)閙ellanox官網(wǎng)沒有scientific linux5.8的相應(yīng)驅(qū)動(dòng),只能下載red hat5.8的操做系統(tǒng)使用源代碼當(dāng)前操做系統(tǒng)編譯安裝

MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.tgz

值得注意的是這個(gè)驅(qū)動(dòng)就算在Red Hat 5.8上安裝最好系統(tǒng)是英文版,最好/usr目錄10GB

(1)解壓到指定目錄,進(jìn)入/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# tar -xzvf MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.tgz -C /usr/src/

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# cd /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

目錄文件以下

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# ls

docsmlnx_add_kernel_support.shrepodatasrc

firmwaremlnxofedinstallRPMSuninstall.sh

(2)標(biāo)準(zhǔn)安裝就報(bào)錯(cuò)了,以下

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# ./mlnxofedinstall The 2.6.18-308.1.1.el5 kernel is installed, MLNX_OFED does not have drivers available for this kernel.

You can run mlnx_add_kernel_support.sh in order to to generate an MLNX_OFED package with drivers for this kernel.

根據(jù)提示添加kernel吧,固然也報(bào)錯(cuò),不詳細(xì)說了,沒辦法只有源代碼了以下:進(jìn)入源代碼src/

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# cd /usr/src/MLNX_O

FED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/

解壓源代碼包src/MLNX_OFED_SRC-1.5.3-4.0.42.tgz

(3)解壓出來的源代碼目錄以下,并執(zhí)行源代碼安裝了

[root@localhost src]# cd MLNX_OFED_SRC-1.5.3-4.0.42

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ls

BUILD_IDinstall.plLICENSEofed.confRPMSSRPMSuninstall.sh

(4)執(zhí)行源代碼安裝腳本

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ./install.pl

OFED Distribution Software Installation Menu

1) View OFED Installation Guide查看菜單

2) Install OFED Software安裝驅(qū)動(dòng)

3) Show Installed Software顯示驅(qū)動(dòng)文件包

4) Configure IPoIB配置IB卡文件

5) Uninstall OFED Software卸載

Q) Exit

Select Option [1-5]:2

OFED Distribution Software Installation Menu

1) Basic (OFED modules and basic user level libraries)基本

2) HPC (OFED modules and libraries, MPI and diagnostic tools)基本+MPI

3) All packages (all of Basic, HPC)全部

4) Customize自定義

5) Packages required by VMA (IB and Eth)

6) Packages required by VMA IB

7) Packages required by VMA Eth

Q) Exit

Select Option [1-7]:3全部

Please choose an implementation of MVAPICH2:

1) OFA (IB and iWARP)

2) uDAPL

Implementation [1]: 1

Enable ROMIO support [Y/n]: y

Enable shared library support [Y/n]: y

Enable Checkpoint-Restart support [y/N]: y

BLCR installation directory [or nothing if not installed]: /usr(當(dāng)上面選擇)

注意:下面是須要安裝的包文件,這個(gè)不是系統(tǒng)自帶的包,是OFED中的軟件包

libmlx4-debuginfo

libmverbs-devel

libmqe

libmqe-devel

libcxgb3

libcxgb3-devel

libcxgb3-debuginfo

libnes

libnes-devel-static

libnes-debuginfo

libipathverbs

libipathverbs-devel

libipathverbs-debuginfo

libibcm

libibcm-devel

libibcm-debuginfo

libibumad

libibumad-devel

libibumad-static

libibumad-debuginfo

libibmad

libibmad-devel

libibmad-static

libibmad-debuginfo

ibsim

ibsim-debuginfo

ibacm

librdmacm

librdmacm-utils

librdmacm-devel

librdmacm-debuginfo

libsdp

libsdp-devel

libsdp-debuginfo

compat-dapl

compat-dapl-devel

dapl

dapl-devel

dapl-devel-static

dapl-utils

dapl-debuginfo

perftest

mstflint

sdpnetstat

srptools

rds-tools

rds-devel

ibutils本次安裝腳本安裝在此處將報(bào)錯(cuò):并終止,后續(xù)需手工安裝

infiniband-diags

qperf

qperf-debuginfo

ofed-docs

openshmem

mpi-selector

openmpi_gcc

bupc

infinipath-psm

infinipath-psm-devel

mvapich2_gcc

mpitests_mvapich2_gcc

mpitests_openmpi_gcc(到這里知道咱們須要安裝那些包,這是咱們手工安裝的依據(jù))

Uninstalling the previous version of OFED編譯安裝過程以下

Install ofed-scripts RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

Build mlnx-ofa_kernel RPM

Running rpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --nodeps --define '_dist .unsupported' --define 'configure_options--with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_ib-mod --with-mlx4_vnic-mod --with-nes-mod --with-qib-mod --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-rds-mod --with-iser-mod' --define 'build_kernel_ib 1' --define 'build_kernel_ib_devel 1' --define 'KVERSION 2.6.18-308.1.1.el5' --define 'K_SRC /lib/modules/2.6.18-308.1.1.el5/build' --define 'network_dir /etc/sysconfig/network-scripts' --define '_prefix /usr' --define '__arch_install_post %{nil}' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.src.rpm

Install mlnx-ofa_kernel RPM:

Installing mlnx-ofa_kernel RPM

cmd: rpm -ivh--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.unsupported.x86_64.rpm

Preparing...##################################################

mlnx-ofa_kernel##################################################

Installing kmod-mlnx-ofa_kernel RPM

Preparing...##################################################

kmod-mlnx-ofa_kernel##################################################

Install mlnx-ofa_kernel-devel RPM:

Installing mlnx-ofa_kernel-devel RPM

cmd: rpm -ivh--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mlnx-ofa_kernel-devel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.unsupported.x86_64.rpm

Preparing...##################################################

mlnx-ofa_kernel-devel##################################################

Build knem RPM

Running rpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define '_release 2.6.18_308.1.1.el5' --define 'KVERSION 2.6.18-308.1.1.el5' --define 'K_SRC /lib/modules/2.6.18-308.1.1.el5/build' --define '_prefix /usr' --define '__arch_install_post %{nil}' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/knem-0.9.7mlnx1-0.src.rpm

Install knem RPM:

Running rpm -iv--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/knem-0.9.7mlnx1-2.6.18_308.1.1.el5.x86_64.rpm

Install kernel-mft RPM:

Running rpm -iv--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/kernel-mft-2.7.1-2.6.18_308.1.1.el5.x86_64.rpm

Install libibverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-devel-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-devel-static RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-devel-static-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-utils-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-debuginfo-1.1.5mlnx2-1.x86_64.rpm

Install libmthca RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmthca-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

Install libmthca-devel-static RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmthca-devel-static-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

Install libmthca-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmthca-debuginfo-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

Install libmverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmverbs-0.1.0-3.15.gd28970e.x86_64.rpm

Install libmlx4 RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmlx4-1.0.2mlnx6-1.x86_64.rpm

Install libmlx4-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmlx4-devel-1.0.2mlnx6-1.x86_64.rpm

Install libmlx4-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmlx4-debuginfo-1.0.2mlnx6-1.x86_64.rpm

Install libmverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmverbs-devel-0.1.0-3.15.gd28970e.x86_64.rpm

Build libmqe RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libmqe-0.3.0-4.9.g973226d.src.rpm

Install libmqe RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmqe-0.3.0-4.9.g973226d.x86_64.rpm

Install libmqe-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmqe-devel-0.3.0-4.9.g973226d.x86_64.rpm

Install libcxgb3 RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libcxgb3-1.3.1-1.x86_64.rpm

Install libcxgb3-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libcxgb3-devel-1.3.1-1.x86_64.rpm

Install libcxgb3-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libcxgb3-debuginfo-1.3.1-1.x86_64.rpm

Install libnes RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libnes-1.1.1mlnx1-1.x86_64.rpm

Install libnes-devel-static RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libnes-devel-static-1.1.1mlnx1-1.x86_64.rpm

Install libnes-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libnes-debuginfo-1.1.1mlnx1-1.x86_64.rpm

Install libipathverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libipathverbs-1.2mlnx1-1.x86_64.rpm

Install libipathverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libipathverbs-devel-1.2mlnx1-1.x86_64.rpm

Install libipathverbs-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libipathverbs-debuginfo-1.2mlnx1-1.x86_64.rpm

Build libibcm RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libibcm-1.0.5mlnx1-1.src.rpm

Install libibcm RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibcm-1.0.5mlnx1-1.x86_64.rpm

Install libibcm-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibcm-devel-1.0.5mlnx1-1.x86_64.rpm

Install libibcm-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibcm-debuginfo-1.0.5mlnx1-1.x86_64.rpm

Build libibumad RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libibumad-1.3.7.MLNX_20130110_ff06102-0.1.src.rpm

Install libibumad RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibumad-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

Install libibumad-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibumad-devel-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

Install libibumad-static RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibumad-static-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

Install libibumad-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibumad-debuginfo-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

Build libibmad RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libibmad-1.3.8.MLNX_20120424-0.1.src.rpm

Install libibmad RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibmad-1.3.8.MLNX_20120424-0.1.x86_64.rpm

Install libibmad-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibmad-devel-1.3.8.MLNX_20120424-0.1.x86_64.rpm

Install libibmad-static RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibmad-static-1.3.8.MLNX_20120424-0.1.x86_64.rpm

Install libibmad-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibmad-debuginfo-1.3.8.MLNX_20120424-0.1.x86_64.rpm

Build ibsim RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ibsim-0.5mlnx1-0.1.g327c3d8.src.rpm

Install ibsim RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ibsim-0.5mlnx1-0.1.g327c3d8.x86_64.rpm

Install ibsim-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ibsim-debuginfo-0.5mlnx1-0.1.g327c3d8.x86_64.rpm

Build ibacm RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ibacm-1.0.4-1.src.rpm

Install ibacm RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ibacm-1.0.4-1.x86_64.rpm

Install librdmacm RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-1.0.15-1.x86_64.rpm

Install librdmacm-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-utils-1.0.15-1.x86_64.rpm

Install librdmacm-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-devel-1.0.15-1.x86_64.rpm

Install librdmacm-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-debuginfo-1.0.15-1.x86_64.rpm

Build libsdp RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/libsdp-1.1.108-0.16.g5cf59e8.src.rpm

Install libsdp RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libsdp-1.1.108-0.16.g5cf59e8.x86_64.rpm

Install libsdp-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libsdp-devel-1.1.108-0.16.g5cf59e8.x86_64.rpm

Install libsdp-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libsdp-debuginfo-1.1.108-0.16.g5cf59e8.x86_64.rpm

Build compat-dapl RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_defaultdocdir /usr/share/doc/compat-dapl-1.2.19mlnx1' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/compat-dapl-1.2.19mlnx1-1.src.rpm

Install compat-dapl RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/compat-dapl-1.2.19mlnx1-1.x86_64.rpm

Install compat-dapl-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/compat-dapl-devel-1.2.19mlnx1-1.x86_64.rpm

Build dapl RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_defaultdocdir /usr/share/doc/dapl-2.0.34' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/dapl-2.0.34-1.src.rpm

Install dapl RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/dapl-2.0.34-1.x86_64.rpm

Install dapl-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/dapl-devel-2.0.34-1.x86_64.rpm

Install dapl-devel-static RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/dapl-devel-static-2.0.34-1.x86_64.rpm

Install dapl-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/dapl-utils-2.0.34-1.x86_64.rpm

Install dapl-debuginfo RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/dapl-debuginfo-2.0.34-1.x86_64.rpm

Build perftest RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/perftest-2.0-0.37.g96050b8.src.rpm

Install perftest RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/perftest-2.0-0.37.g96050b8.x86_64.rpm

Install mstflint RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mstflint-1.4mlnx4-1.21.gd948ddd.x86_64.rpm

Build sdpnetstat RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/sdpnetstat-1.60-0.3.gb6e7425.src.rpm

Install sdpnetstat RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/sdpnetstat-1.60-0.3.gb6e7425.x86_64.rpm

Build srptools RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/srptools-0.0.4mlnx3-1.src.rpm

Install srptools RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/srptools-0.0.4mlnx3-1.x86_64.rpm

Build rds-tools RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/rds-tools-2.0.4mlnx1-1.src.rpm

Install rds-tools RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/rds-tools-2.0.4mlnx1-1.x86_64.rpm

Install rds-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/rds-devel-2.0.4mlnx1-1.x86_64.rpm

Build ibutils RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /opt/ibutils' --define '_exec_prefix /opt/ibutils' --define '_sysconfdir /etc' --define '_usr /opt/ibutils' --define '_mandir /opt/ibutils/share/man' --define 'build_ibmgtsim 1' --define '__arch_install_post %{nil}' --define 'configure_options--with-osm=/usr ' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ibutils-1.5.7mlnx2-0.2.g05a9d1a.src.rpm

Failed to build ibutils RPM報(bào)錯(cuò)build失敗

See /tmp/OFED.17106.logs/ibutils.rpmbuild.log

5.查看報(bào)錯(cuò)信息

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# cat/tmp/OFED.17106.logs/ibutils.rpmbuild.log

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /opt/ibutils' --define '_exec_prefix /opt/ibutils' --define '_sysconfdir /etc' --define '_usr /opt/ibutils' --define '_mandir /opt/ibutils/share/man' --define 'build_ibmgtsim 1' --define '__arch_install_post %{nil}' --define 'configure_options--with-osm=/usr ' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ibutils-1.5.7mlnx2-0.2.g05a9d1a.src.rpm

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

error: Failed build dependencies:

opensm-devel is needed by ibutils-1.5.7mlnx2-0.2.g05a9d1a.x86_64

Installing /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ibutils-1.5.7mlnx2-0.2.g05a9d1a.src.rpm

Building target platforms: x86_64

Building for target x86_64

看到報(bào)錯(cuò)日志可知道:(1)ibutils-1.5.7mlnx2-0.2.g05a9d1a.src.rpm軟件build過程失敗

(2)提示的一些沒有用戶使用root執(zhí)行警告信息

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

(3)依賴關(guān)系:opensm-devel is needed by ibutils-1.5.7mlnx2-0.2.g05a9d1a.x86_64

(4)手工解決下面沒安裝的

ibutils

infiniband-diags

qperf

qperf-debuginfo

ofed-docs

openshmem

mpi-selector

openmpi_gcc

bupc

infinipath-psm

infinipath-psm-devel

mvapich2_gcc

mpitests_mvapich2_gcc

mpitests_openmpi_gcc

看到ibutils這個(gè)包安裝須要依賴opensm-devel,值得注意的是這個(gè)包不是系統(tǒng)自帶的,是OFED驅(qū)動(dòng)中的opensm-devel-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

6.進(jìn)入OFED驅(qū)動(dòng)文件的RPM源代碼包,并執(zhí)行編譯

cd /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# cd SRPMS/

[root@localhost SRPMS]# ls

bupc-2.2-370.src.rpm

compat-dapl-1.2.19mlnx1-1.src.rpm

dapl-2.0.34-1.src.rpm

ibacm-1.0.4-1.src.rpm

ib-bonding-0.9.0-43.src.rpm

ibsim-0.5mlnx1-0.1.g327c3d8.src.rpm

ibutils-1.5.7mlnx2-0.2.g05a9d1a.src.rpm

infiniband-diags-1.5.13.MLNX_20121224-0.1.src.rpm

infinipath-psm-2.9mlnx1-926.1005_open.src.rpm

kernel-mft-2.7.1-7.src.rpm

knem-0.9.7mlnx1-0.src.rpm

libcxgb3-1.3.1-1.src.rpm

libehca-1.2.2-0.1.g69e1a88.src.rpm

libibcm-1.0.5mlnx1-1.src.rpm

libibmad-1.3.8.MLNX_20120424-0.1.src.rpm

libibumad-1.3.7.MLNX_20130110_ff06102-0.1.src.rpm

libibverbs-1.1.5mlnx2-1.src.rpm

libipathverbs-1.2mlnx1-1.src.rpm

libmlx4-1.0.2mlnx6-1.src.rpm

libmqe-0.3.0-4.9.g973226d.src.rpm

libmthca-1.0.6mlnx1-0.1.gbe5eef3.src.rpm

libmverbs-0.1.0-3.15.gd28970e.src.rpm

libnes-1.1.1mlnx1-1.src.rpm

librdmacm-1.0.15-1.src.rpm

libsdp-1.1.108-0.16.g5cf59e8.src.rpm

mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.src.rpm

mpi-selector-1.0.3-1.src.rpm

mpitests-3.2-923.src.rpm

mstflint-1.4mlnx4-1.21.gd948ddd.src.rpm

mvapich-1.2.0-3635.src.rpm

mvapich2-1.9a-1.src.rpm

ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.src.rpm

ofa_kernel_dkms-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.src.rpm

ofed-docs-1.5.3-OFED.1.5.3.4.0.42.src.rpm

ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.src.rpm

openmpi-1.6.4-1.src.rpm

openshmem-2.2-23217.src.rpm

perftest-2.0-0.37.g96050b8.src.rpm

qlvnictools-0.0.1-0.1.ge27eef7.src.rpm

qperf-0.4.6mlnx1-0.1.gb81434e.src.rpm

rds-tools-2.0.4mlnx1-1.src.rpm

sdpnetstat-1.60-0.3.gb6e7425.src.rpm

srptools-0.0.4mlnx3-1.src.rpm

[root@localhost SRPMS]# rpm -ivh ./*

1:bupcwarning: user hpcuser does not exist - using root

warning: group ecryptfs does not exist - using root提示信息

########################################### [100%]

warning: user hpcuser does not exist - using root

warning: group ecryptfs does not exist - using root

2:compat-daplwarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

3:daplwarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

warning: user vlad does not exist - using root99%)

warning: group ssl-cert does not exist - using root

########################################### [100%]

4:ibacmwarning: user vlad does not exist - using root

warning: group vlad does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group vlad does not exist - using root

5:ib-bondingwarning: user monis does not exist - using root5%)

warning: group messagebus does not exist - using root

########################################### [100%]

6:ibsimwarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

warning: user vlad does not exist - using root97%)

warning: group netdev does not exist - using root

########################################### [100%]

7:ibutilswarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

8:infiniband-diagswarning: group nogroup does not exist - using root

warning: user builder does not exist - using root)

warning: group libuuid does not exist - using root

########################################### [100%]

9:infinipath-psmwarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

10:kernel-mftwarning: user mohammad does not exist - using root

warning: group mtl does not exist - using root

warning: user mohammad does not exist - using root

warning: group mtl does not exist - using root

########################################### [100%]

11:knemwarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

12:libcxgb3warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

13:libehcawarning: user vlad does not exist - using root

warning: group vlad does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group vlad does not exist - using root

14:libibcmwarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

15:libibmadwarning: user builder does not exist - using root

warning: group libuuid does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

16:libibumadwarning: group nogroup does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

17:libibverbswarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

18:libipathverbswarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

19:libmlx4warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

20:libmqewarning: user marcela does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group libuuid does not exist - using root

21:libmthcawarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

22:libmverbswarning: user vlad does not exist - using root

warning: group libuuid does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

23:libneswarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

24:librdmacmwarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

25:libsdpwarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

26:mlnx-ofa_kernelwarning: user builder does not exist - using root

warning: group libuuid does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

27:mpi-selectorwarning: user vlad does not exist - using root

warning: group vlad does not exist - using root

warning: user vlad does not exist - using root89%)

warning: group vlad does not exist - using root

########################################### [100%]

28:mpitestswarning: user pasha does not exist - using root

warning: group messagebus does not exist - using root

########################################### [100%]

warning: user pasha does not exist - using root

warning: group messagebus does not exist - using root

29:mstflintwarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

30:mvapich########################################### [100%]

31:mvapich2warning: user hpcuser does not exist - using root

warning: group mtl does not exist - using root

########################################### [100%]

warning: user hpcuser does not exist - using root

warning: group mtl does not exist - using root

32:ofa_kernelwarning: user builder does not exist - using root

warning: group libuuid does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

33:ofa_kernel_dkmswarning: user builder does not exist - using root

warning: group libuuid does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

34:ofed-docswarning: user builder does not exist - using root

warning: group libuuid does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

35:ofed-scriptswarning: user builder does not exist - using root

warning: group libuuid does not exist - using root

warning: user builder does not exist - using root)

warning: group libuuid does not exist - using root

########################################### [100%]

36:openmpiwarning: user hpcuser does not exist - using root

warning: group mtl does not exist - using root

warning: user hpcuser does not exist - using root

warning: group mtl does not exist - using root

########################################### [100%]

37:openshmemwarning: user hpcuser does not exist - using root

warning: group ecryptfs does not exist - using root

########################################### [100%]

warning: user hpcuser does not exist - using root

warning: group ecryptfs does not exist - using root

38:perftestwarning: user idos does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user builder does not exist - using root

warning: group libuuid does not exist - using root

39:qlvnictoolswarning: user vlad does not exist - using root

warning: group vlad does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group vlad does not exist - using root

40:qperfwarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group netdev does not exist - using root

41:rds-toolswarning: user vlad does not exist - using root

warning: group netdev does not exist - using root

warning: user vlad does not exist - using root98%)

warning: group netdev does not exist - using root

########################################### [100%]

42:sdpnetstatwarning: user vlad does not exist - using root

warning: group vlad does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group vlad does not exist - using root

43:srptoolswarning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

########################################### [100%]

warning: user vlad does not exist - using root

warning: group ssl-cert does not exist - using root

7.解決上面用戶警告:

useradd-s /sbin/nologin -M does

useradd-s /sbin/nologin -M vlad

useradd-s /sbin/nologin -M netdev

useradd-s /sbin/nologin -M libuuid

useradd-s /sbin/nologin -M builder

useradd-s /sbin/nologin -M ecryptfs

useradd-s /sbin/nologin -M hpcuser

useradd-s /sbin/nologin -M mtl

useradd-s /sbin/nologin -M messagebus

useradd-s /sbin/nologin -M pasha

useradd-s /sbin/nologin -M mohammad

useradd-s /sbin/nologin -M nogroup

useradd-s /sbin/nologin -M ssl-cert

useradd-s /sbin/nologin -M idos

useradd-s /sbin/nologin -Mmarcela

useradd-s /sbin/nologin -Mmonis

[root@localhost SRPMS]# rpm -ivh ./*#注意這里不是安裝RPM驅(qū)動(dòng)包,這個(gè)是根據(jù)源代碼包,生成buitd的文件

1:bupc########################################### [100%]

2:compat-dapl########################################### [100%]

3:dapl########################################### [100%]

4:ibacm########################################### [100%]

5:ib-bonding########################################### [100%]

6:ibsim########################################### [100%]

7:ibutils########################################### [100%]

8:infiniband-diags########################################### [100%]

9:infinipath-psm########################################### [100%]

10:kernel-mft########################################### [100%]

11:knem########################################### [100%]

12:libcxgb3########################################### [100%]

13:libehca########################################### [100%]

14:libibcm########################################### [100%]

15:libibmad########################################### [100%]

16:libibumad########################################### [100%]

17:libibverbs########################################### [100%]

18:libipathverbs########################################### [100%]

19:libmlx4########################################### [100%]

20:libmqe########################################### [100%]

21:libmthca########################################### [100%]

22:libmverbs########################################### [100%]

23:libnes########################################### [100%]

24:librdmacm########################################### [100%]

25:libsdp########################################### [100%]

26:mlnx-ofa_kernel########################################### [100%]

27:mpi-selector########################################### [100%]

28:mpitests########################################### [100%]

29:mstflint########################################### [100%]

30:mvapich########################################### [100%]

31:mvapich2########################################### [100%]

32:ofa_kernel########################################### [100%]

33:ofa_kernel_dkms########################################### [100%]

34:ofed-docs########################################### [100%]

35:ofed-scripts########################################### [100%]

36:openmpi########################################### [100%]

37:openshmem########################################### [100%]

38:perftest########################################### [100%]

39:qlvnictools########################################### [100%]

40:qperf########################################### [100%]

41:rds-tools########################################### [100%]

42:sdpnetstat########################################### [100%]

43:srptools########################################### [100%]

用戶警告解決了,繼續(xù)

7.進(jìn)入build的文件目錄cd /usr/src/redhat/SPECS/以下,build的opensm-devel.spec,發(fā)現(xiàn)么有。坑爹呀

[root@localhost RPMS]# cd /usr/src/redhat/SPECS/

[root@localhost SPECS]# ls

bupc.speclibibverbs.specmvapich2.spec

dapl.speclibipathverbs.specmvapich.spec

ibacm.speclibmlx4.specofa_kernel_dkms.spec

ib-bonding.speclibmqe.specofa_kernel.spec

ibsim.speclibmthca.specofed-docs.spec

ibutils.speclibmverbs.specofed-scripts.spec

infiniband-diags.speclibnes.specopenmpi-1.6.4.spec

infinipath-psm.speclibrdmacm.specoshmem.spec

knem.speclibsdp.specperftest.spec

libcxgb3.specmft_kernel.specqlvnictools.spec

libehca.specmlnx-ofa_kernel.specqperf.spec

libibcm.specmpi-selector.specrds-tools.spec

libibmad.specmpitests.specsdpnetstat.spec

libibumad.specmstflint.specsrptools.spec

8.進(jìn)入/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

RPMS/目錄,看看打包好的有么有了

[root@localhost RPMS]# ls

ar_mgr-1.0-0.8.gc8aa6b2.x86_64.rpm

bupc-2.2-370.x86_64.rpm

cc_mgr-1.0-0.7.g5cbb98a.x86_64.rpm

compat-dapl-1.2.19mlnx1-1.i686.rpm

compat-dapl-1.2.19mlnx1-1.x86_64.rpm

compat-dapl-devel-1.2.19mlnx1-1.i686.rpm

compat-dapl-devel-1.2.19mlnx1-1.x86_64.rpm

compat-dapl-devel-static-1.2.19mlnx1-1.i686.rpm

compat-dapl-devel-static-1.2.19mlnx1-1.x86_64.rpm

compat-dapl-utils-1.2.19mlnx1-1.i686.rpm

compat-dapl-utils-1.2.19mlnx1-1.x86_64.rpm

dapl-2.0.34-1.i686.rpm

dapl-2.0.34-1.x86_64.rpm

dapl-devel-2.0.34-1.i686.rpm

dapl-devel-2.0.34-1.x86_64.rpm

dapl-devel-static-2.0.34-1.i686.rpm

dapl-devel-static-2.0.34-1.x86_64.rpm

dapl-utils-2.0.34-1.i686.rpm

dapl-utils-2.0.34-1.x86_64.rpm

dump_pr-1.0-0.1.g79715a9.x86_64.rpm

fca-2.5.15cd296-1.x86_64.rpm

ibacm-1.0.4-1.x86_64.rpm

ibdump-1.0.6-1.x86_64.rpm

ibsim-0.5mlnx1-0.1.g327c3d8.x86_64.rpm

ibutils-1.5.7mlnx2-0.2.g05a9d1a.x86_64.rpm

ibutils2-2.1.1-0.32.g63f2b32.x86_64.rpm

infiniband-diags-1.5.13.MLNX_20121224-0.1.x86_64.rpm

infiniband-diags-guest-1.5.13.MLNX_20121224-0.1.x86_64.rpm

infinipath-psm-2.9mlnx1-926.1005_open.x86_64.rpm

infinipath-psm-devel-2.9mlnx1-926.1005_open.x86_64.rpm

kernel-mft-2.7.1-2.6.18_308.el5.x86_64.rpm

kernel-mft-2.7.1-2.6.18_308.el5xen.x86_64.rpm

kmod-mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8.x86_64.rpm

kmod-mlnx-ofa_kernel-xen-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8.x86_64.rpm

knem-0.9.7mlnx1-2.6.18_308.el5.x86_64.rpm

knem-0.9.7mlnx1-2.6.18_308.el5xen.x86_64.rpm

libcxgb3-1.3.1-1.i686.rpm

libcxgb3-1.3.1-1.x86_64.rpm

libcxgb3-devel-1.3.1-1.i686.rpm

libcxgb3-devel-1.3.1-1.x86_64.rpm

libibcm-1.0.5mlnx1-1.i686.rpm

libibcm-1.0.5mlnx1-1.x86_64.rpm

libibcm-devel-1.0.5mlnx1-1.i686.rpm

libibcm-devel-1.0.5mlnx1-1.x86_64.rpm

libibmad-1.3.8.MLNX_20120424-0.1.i686.rpm

libibmad-1.3.8.MLNX_20120424-0.1.x86_64.rpm

libibmad-devel-1.3.8.MLNX_20120424-0.1.i686.rpm

libibmad-devel-1.3.8.MLNX_20120424-0.1.x86_64.rpm

libibmad-static-1.3.8.MLNX_20120424-0.1.i686.rpm

libibmad-static-1.3.8.MLNX_20120424-0.1.x86_64.rpm

libibumad-1.3.7.MLNX_20130110_ff06102-0.1.i686.rpm

libibumad-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

libibumad-devel-1.3.7.MLNX_20130110_ff06102-0.1.i686.rpm

libibumad-devel-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

libibumad-static-1.3.7.MLNX_20130110_ff06102-0.1.i686.rpm

libibumad-static-1.3.7.MLNX_20130110_ff06102-0.1.x86_64.rpm

libibverbs-1.1.5mlnx2-1.i686.rpm

libibverbs-1.1.5mlnx2-1.x86_64.rpm

libibverbs-devel-1.1.5mlnx2-1.i686.rpm

libibverbs-devel-1.1.5mlnx2-1.x86_64.rpm

libibverbs-devel-static-1.1.5mlnx2-1.i686.rpm

libibverbs-devel-static-1.1.5mlnx2-1.x86_64.rpm

libibverbs-utils-1.1.5mlnx2-1.i686.rpm

libibverbs-utils-1.1.5mlnx2-1.x86_64.rpm

libipathverbs-1.2mlnx1-1.i686.rpm

libipathverbs-1.2mlnx1-1.x86_64.rpm

libipathverbs-devel-1.2mlnx1-1.i686.rpm

libipathverbs-devel-1.2mlnx1-1.x86_64.rpm

libmlx4-1.0.2mlnx6-1.i686.rpm

libmlx4-1.0.2mlnx6-1.x86_64.rpm

libmlx4-devel-1.0.2mlnx6-1.i686.rpm

libmlx4-devel-1.0.2mlnx6-1.x86_64.rpm

libmqe-0.3.0-4.9.g973226d.i686.rpm

libmqe-0.3.0-4.9.g973226d.x86_64.rpm

libmqe-devel-0.3.0-4.9.g973226d.i686.rpm

libmqe-devel-0.3.0-4.9.g973226d.x86_64.rpm

libmthca-1.0.6mlnx1-0.1.gbe5eef3.i686.rpm

libmthca-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

libmthca-devel-static-1.0.6mlnx1-0.1.gbe5eef3.i686.rpm

libmthca-devel-static-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

libmverbs-0.1.0-3.15.gd28970e.i686.rpm

libmverbs-0.1.0-3.15.gd28970e.x86_64.rpm

libmverbs-devel-0.1.0-3.15.gd28970e.i686.rpm

libmverbs-devel-0.1.0-3.15.gd28970e.x86_64.rpm

libnes-1.1.1mlnx1-1.i686.rpm

libnes-1.1.1mlnx1-1.x86_64.rpm

libnes-devel-static-1.1.1mlnx1-1.i686.rpm

libnes-devel-static-1.1.1mlnx1-1.x86_64.rpm

librdmacm-1.0.15-1.i686.rpm

librdmacm-1.0.15-1.x86_64.rpm

librdmacm-devel-1.0.15-1.i686.rpm

librdmacm-devel-1.0.15-1.x86_64.rpm

librdmacm-utils-1.0.15-1.i686.rpm

librdmacm-utils-1.0.15-1.x86_64.rpm

libsdp-1.1.108-0.16.g5cf59e8.i686.rpm

libsdp-1.1.108-0.16.g5cf59e8.x86_64.rpm

libsdp-devel-1.1.108-0.16.g5cf59e8.i686.rpm

libsdp-devel-1.1.108-0.16.g5cf59e8.x86_64.rpm

mft-2.7.1a-1.x86_64.rpm

mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8.x86_64.rpm

mlnx-ofa_kernel-devel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.rhel5u8.x86_64.rpm

mlnxofed-docs-1.5.3-4.0.42.noarch.rpm

mpi-selector-1.0.3-1.x86_64.rpm

mpitests_mvapich2_gcc-3.2-923.x86_64.rpm

mpitests_mvapich_gcc-3.2-923.x86_64.rpm

mpitests_openmpi_gcc-3.2-923.x86_64.rpm

mstflint-1.4mlnx4-1.21.gd948ddd.x86_64.rpm

mvapich2_gcc-1.9a-1.x86_64.rpm

mvapich_gcc-1.2.0-3635.x86_64.rpm

mxm-1.5.8f1b855-1.x86_64.rpm

ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

openmpi_gcc-1.6.4-1.x86_64.rpm

openshmem-2.2-23217.x86_64.rpm

opensm-3.3.13.MLNX_20130110_cd124d3-0.1.i686.rpm

opensm-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

opensm-devel-3.3.13.MLNX_20130110_cd124d3-0.1.i686.rpm

opensm-devel-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm還好有啦

opensm-libs-3.3.13.MLNX_20130110_cd124d3-0.1.i686.rpm

opensm-libs-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

opensm-static-3.3.13.MLNX_20130110_cd124d3-0.1.i686.rpm

opensm-static-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

perftest-2.0-0.37.g96050b8.x86_64.rpm

qperf-0.4.6mlnx1-0.1.gb81434e.x86_64.rpm

rds-devel-2.0.4mlnx1-1.x86_64.rpm

rds-tools-2.0.4mlnx1-1.x86_64.rpm

sdpnetstat-1.60-0.3.gb6e7425.x86_64.rpm

srptools-0.0.4mlnx3-1.x86_64.rpm

9.手工安裝腳本安裝沒完成的軟件包

(1)安裝opensm-libs-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

[root@localhost RPMS]# rpm –ivh opensm-libs-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

Preparing...########################################### [100%]

1:opensm-libs########################################### [100%]

(2)在此安裝:opensm-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/opensm-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

Preparing...########################################### [100%]

1:opensm########################################### [100%]

(3)安裝opensm-devel-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

[root@localhost RPMS]# rpm -ivh opensm-devel-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

Preparing...########################################### [100%]

1:opensm-devel########################################### [100%]

(4)安裝opensm-static-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/RPMS/opensm-static-3.3.13.MLNX_20130110_cd124d3-0.1.x86_64.rpm

Preparing...########################################### [100%]

1:opensm-static########################################### [100%]

(5)安裝libibmad-1.3.8.MLNX_20120424-0.1.i686.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/RPMS/libibmad-1.3.8.MLNX_20120424-0.1.i686.rpm

Preparing...########################################### [100%]

1:libibmad########################################### [100%]

(6)安裝libibmad-devel-1.3.8.MLNX_20120424-0.1.i686.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/libibmad-devel-1.3.8.MLNX_20120424-0.1.i686.rpm Preparing...########################################### [100%]

1:libibmad-devel########################################### [100%]

(7)安裝libibmad-static-1.3.8.MLNX_20120424-0.1.i686.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/libibmad-static-1.3.8.MLNX_20120424-0.1.i686.rpm

Preparing...########################################### [100%]

1:libibmad-static########################################### [100%]

(8)安裝libibumad-1.3.7.MLNX_20130110_ff06102-0.1.i686.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/libibumad-1.3.7.MLNX_20130110_ff06102-0.1.i686.rpm

Preparing...########################################### [100%]

1:libibumad########################################### [100%]

(9)安裝libibumad-devel-1.3.7.MLNX_20130110_ff06102-0.1.

i686.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/libibumad-devel-1.3.7.MLNX_20130110_ff06102-0.1.

i686.rpm

Preparing...########################################### [100%]

1:libibumad-devel########################################### [100%]

(10)安裝libibumad-static-1.3.7.MLNX_20130110_ff06102-0.1

.i686.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/libibumad-static-1.3.7.MLNX_20130110_ff06102-0.1

.i686.rpm

Preparing...########################################### [100%]

1:libibumad-static########################################### [100%]

(11)安裝ibutils-1.5.7mlnx2-0.2.g05a9d1a.x86_64.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/ibutils-1.5.7mlnx2-0.2.g05a9d1a.x86_64.rpm

Preparing...########################################### [100%]

1:ibutils########################################### [100%]

(12)安裝ibutils2-2.1.1-0.32.g63f2b32.x86_64.rpm

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1

.5.3-4.0.42-rhel5.8-x86_64/RPMS/ibutils2-2.1.1-0.32.g63f2b32.x86_64.rpm

Preparing...########################################### [100%]

1:ibutils2########################################### [100%]

(13)build infiniband-diags.spec,后生成3個(gè)rpm安裝文件,以下

(注意下面全部build文件生成文件路徑在/usr/src/redhat/RPMS/x86_64/)

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS/infiniband-diags.spec

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/redhat/RPMS/x86_64/infiniband-diags-*

Preparing...########################################### [100%]

1:infiniband-diags-guest ########################################### [ 33%]

2:infiniband-diags########################################### [ 67%]

3:infiniband-diags-debugi########################################### [100%]

(14)bulid的qperf.spec,并安裝生成的rpm包

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS

/qperf.spec

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/redhat/RPMS/x86_

64/qperf-*

Preparing...########################################### [100%]

1:qperf########################################### [ 50%]

2:qperf-debuginfo########################################### [100%]

(15)bulid的ofed-docs.spec,并安裝生成的rpm包

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS/ofed-docs.spec

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/redhat/RPMS/x86_

64/ofed-docs-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

Preparing...########################################### [100%]

1:ofed-docs########################################### [100%]

(16)build的oshmem.spec,并安裝生成的rpm包

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS/oshmem.spec

error: Failed build dependencies:報(bào)錯(cuò)提示:

fca is needed by openshmem-2.2-23217.x86_64須要安裝fca

mxm is needed by openshmem-2.2-23217.x86_64須要安裝mxm

(17)安裝剛才提示的須要安裝的fca,mxm包

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-

1.5.3-4.0.42-rhel5.8-x86_64/RPMS/mxm-1.5.8f1b855-1.x86_64.rpm

Preparing...########################################### [100%]

1:mxm########################################### [100%]

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/MLNX_OFED_LINUX-

1.5.3-4.0.42-rhel5.8-x86_64/RPMS/fca-2.5.15cd296-1.x86_64.rpm

Preparing...########################################### [100%]

1:fca########################################### [100%]

INFO: updating ...

IMPORTANT NOTE:

===============

- The FCA Manager and FCA MPI Runtime library are installed in /opt/mellanox/fca directory.

- The FCA Manager will not be started automatically.

- To start FCA Manager now, type:

/etc/init.d/fca_managerd start

- There should be single process of FCA Manager running per fabric.

- To start FCA Manager automatically after boot, type:

/etc/init.d/fca_managerd install_service

- Check /opt/mellanox/fca/share/doc/fca/README.txt for quick start instructions.

(18)繼續(xù)build oshmem.spec,并安裝

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS/oshmem.spec

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/redhat/RPMS/x86_64/openshmem-2.2-23217.x86_64.rpm

Preparing...########################################### [100%]

1:openshmem########################################### [100%]

(19)build的mpi-selector.spec,并安裝

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS/mpi-selector.spec

root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/redhat/RPMS/x86_

64/mpi-selector-1.0.3-1.x86_64.rpm

Preparing...########################################### [100%]

1:mpi-selector########################################### [100%]

(20)build的openmpi-1.6.4.spec,并安裝

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpmbuild -bb /usr/src/redhat/SPECS/openmpi-1.6.4.spec

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# rpm -ivh /usr/src/redhat/RPMS/x86_

64/openmpi-1.6.4-1.x86_64.rpm

Preparing...########################################### [100%]

1:openmpi########################################### [100%

(20)build的bupc.spec,并安裝

root@localhost x86_64]# rpmbuild -bb /usr/src/redhat/SPECS/bupc.spec

[root@localhost x86_64]# rpm -ivh /usr/src/redhat/RPMS/x86_64/bupc-2.2-370.x86_64.rpm

Preparing...########################################### [100%]

1:bupc########################################### [100%]

(21)bulid的infinipath-psm.spec,并安裝

[root@localhost x86_64]# rpmbuild -bb /usr/src/redhat/SPECS/infinipath-psm.spec

[root@localhost x86_64]# rpm -ivh/usr/src/redhat/RPMS/x86_64/infinipath-psm-*

Preparing...########################################### [100%]

1:infinipath-psm########################################### [ 33%]

2:infinipath-psm-debuginf########################################### [ 67%]

3:infinipath-psm-devel########################################### [100%]

(22)下面文件直接在/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/RPMS/目錄下,安裝便可

[root@localhost x86_64]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/RPMS/mvapich2_gcc-1.9a-1.x86_64.rpm

Preparing...########################################### [100%]

1:mvapich2_gcc########################################### [100%]

[root@localhost x86_64]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8

-x86_64/RPMS/mvapich_gcc-1.2.0-3635.x86_64.rpm

Preparing...########################################### [100%]

1:mvapich_gcc########################################### [100%]

[root@localhost x86_64]# rpm -ivh /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/RPMS/mpitests_*

Preparing...########################################### [100%]

1:mpitests_openmpi_gcc########################################### [ 33%]

2:mpitests_mvapich2_gcc########################################### [ 67%]

3:mpitests_mvapich_gcc########################################### [100%]

10.設(shè)置開機(jī)啟動(dòng),與Linux經(jīng)常使用服務(wù)設(shè)置同樣

(1)手工啟動(dòng),查看服務(wù)是否正常

[root@localhost x86_64]# /etc/init.d/openibd restart

Unloading HCA driver:[OK]

Loading HCA driver and Access Layer:[OK]

Setting up InfiniBand network interfaces:

Bringing up interface ib0:[OK]

Bringing up interface ib1:[OK]

Setting up service network . . .[done]

(2)設(shè)置開機(jī)啟動(dòng)

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# chkconfig --level 5 openibd on設(shè)置開啟啟動(dòng)服務(wù)

11.配置IB卡文件ifcfg-ib0,同網(wǎng)卡配置同樣,

12設(shè)置開機(jī)啟動(dòng)opensm服務(wù),與Linux經(jīng)常使用服務(wù)設(shè)置同樣

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# chkconfig --level 5 opensm on

13.查看IB卡,同時(shí)能夠Ping測(cè)試。

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 1

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... OFED-internal-1.5.3-4.0.42: 2.6.18-308.1.1.el5

Host Driver RPM Check .................. PASS

Firmware on CA #0 VPI .................. v2.11.550

Firmware Check on CA #0 (VPI) .......... NA

REASON: NO required fw version

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 1

總結(jié)

以上是生活随笔為你收集整理的linux安装软件命令1003无标题,linux系统安装OFED(infiniband)的全部內(nèi)容,希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò),歡迎將生活随笔推薦給好友。

亚洲中文字幕无码一久久区 | 国产成人人人97超碰超爽8 | 亚洲一区二区三区香蕉 | 人妻少妇精品无码专区二区 | 精品国产精品久久一区免费式 | 色爱情人网站 | 久久久中文久久久无码 | 久久久婷婷五月亚洲97号色 | 国产精品手机免费 | 国产内射爽爽大片视频社区在线 | 亚洲 a v无 码免 费 成 人 a v | 久久久精品人妻久久影视 | 成熟妇人a片免费看网站 | 99久久婷婷国产综合精品青草免费 | 久久久亚洲欧洲日产国码αv | 国语精品一区二区三区 | 欧美xxxxx精品 | 日韩精品无码免费一区二区三区 | 麻豆国产97在线 | 欧洲 | 亚洲一区二区三区无码久久 | 久久久久国色av免费观看性色 | 婷婷丁香六月激情综合啪 | 乱人伦中文视频在线观看 | 天堂在线观看www | 又大又硬又黄的免费视频 | www一区二区www免费 | 日本在线高清不卡免费播放 | 强奷人妻日本中文字幕 | 国产免费无码一区二区视频 | 久久国内精品自在自线 | 人妻少妇精品视频专区 | 国产亚洲精品久久久久久国模美 | 日本肉体xxxx裸交 | 在线天堂新版最新版在线8 | 精品熟女少妇av免费观看 | 久久久久久久人妻无码中文字幕爆 | 日韩精品无码一本二本三本色 | 欧美猛少妇色xxxxx | 亚洲七七久久桃花影院 | 荫蒂添的好舒服视频囗交 | 日本www一道久久久免费榴莲 | 久久久无码中文字幕久... | 久久亚洲国产成人精品性色 | 亚洲欧美综合区丁香五月小说 | 亚洲综合久久一区二区 | 蜜桃av蜜臀av色欲av麻 999久久久国产精品消防器材 | 国产无av码在线观看 | 樱花草在线播放免费中文 | 久久zyz资源站无码中文动漫 | 国产偷国产偷精品高清尤物 | 在线观看欧美一区二区三区 | 在线看片无码永久免费视频 | 青青草原综合久久大伊人精品 | 欧美猛少妇色xxxxx | 性做久久久久久久免费看 | 国产无套粉嫩白浆在线 | 伊人久久大香线蕉av一区二区 | 亚洲成色www久久网站 | 国产激情无码一区二区app | 免费无码午夜福利片69 | 国产精品怡红院永久免费 | 亚洲人成网站色7799 | 久在线观看福利视频 | 中文无码精品a∨在线观看不卡 | 日本丰满熟妇videos | 樱花草在线播放免费中文 | 日本乱人伦片中文三区 | 国产成人精品一区二区在线小狼 | 国产精品第一区揄拍无码 | 久激情内射婷内射蜜桃人妖 | 国产精品内射视频免费 | 日欧一片内射va在线影院 | 中文字幕人妻丝袜二区 | 免费无码一区二区三区蜜桃大 | 亚洲精品一区国产 | 青草视频在线播放 | 国产亚洲精品久久久久久久 | 老太婆性杂交欧美肥老太 | 亚洲综合无码久久精品综合 | 亚洲成色www久久网站 | 国产亚洲精品久久久ai换 | 丰满护士巨好爽好大乳 | 国产精品沙发午睡系列 | 曰韩无码二三区中文字幕 | 色婷婷欧美在线播放内射 | 欧美国产日韩亚洲中文 | 精品一二三区久久aaa片 | 人妻少妇精品无码专区动漫 | 国产精品久久久久无码av色戒 | 少妇人妻av毛片在线看 | 小鲜肉自慰网站xnxx | 国产综合久久久久鬼色 | 国产成人精品一区二区在线小狼 | 国产特级毛片aaaaaaa高清 | 精品偷拍一区二区三区在线看 | 日韩人妻无码一区二区三区久久99 | 国产精品久久久久9999小说 | 精品国偷自产在线 | 国产精品.xx视频.xxtv | 亚洲va欧美va天堂v国产综合 | 人人妻人人澡人人爽欧美精品 | 成人一在线视频日韩国产 | 亚洲欧美综合区丁香五月小说 | 精品乱子伦一区二区三区 | 国产 浪潮av性色四虎 | 性生交大片免费看女人按摩摩 | 中文字幕久久久久人妻 | 任你躁国产自任一区二区三区 | 亚洲 欧美 激情 小说 另类 | 国产av无码专区亚洲awww | 中文字幕av日韩精品一区二区 | 亚洲欧洲日本综合aⅴ在线 | 日本乱人伦片中文三区 | 国产熟妇另类久久久久 | 亚洲精品欧美二区三区中文字幕 | √8天堂资源地址中文在线 | 麻豆成人精品国产免费 | 无码av中文字幕免费放 | 国产精品无码成人午夜电影 | 日本一卡2卡3卡四卡精品网站 | 国产激情综合五月久久 | 人妻体内射精一区二区三四 | 国产熟女一区二区三区四区五区 | 一本色道久久综合亚洲精品不卡 | 丰满少妇女裸体bbw | 久久综合给合久久狠狠狠97色 | 欧美午夜特黄aaaaaa片 | 久久久久久a亚洲欧洲av冫 | 亚洲人成影院在线观看 | 亚洲爆乳大丰满无码专区 | 国产深夜福利视频在线 | 岛国片人妻三上悠亚 | 亚洲啪av永久无码精品放毛片 | 天天做天天爱天天爽综合网 | 亚洲色偷偷偷综合网 | 久热国产vs视频在线观看 | 欧美精品一区二区精品久久 | 一本久道久久综合婷婷五月 | 国产婷婷色一区二区三区在线 | 综合网日日天干夜夜久久 | 牲欲强的熟妇农村老妇女 | 激情内射亚州一区二区三区爱妻 | 偷窥日本少妇撒尿chinese | 亚洲综合无码久久精品综合 | 中文字幕+乱码+中文字幕一区 | v一区无码内射国产 | 国产极品视觉盛宴 | 男女猛烈xx00免费视频试看 | 国产精品欧美成人 | 特级做a爰片毛片免费69 | 夜夜躁日日躁狠狠久久av | 99精品国产综合久久久久五月天 | 日韩精品成人一区二区三区 | аⅴ资源天堂资源库在线 | 人妻少妇精品无码专区动漫 | 高清不卡一区二区三区 | 精品欧洲av无码一区二区三区 | 狠狠色噜噜狠狠狠7777奇米 | 亚洲国产精品无码一区二区三区 | 青草视频在线播放 | 亚洲精品国产品国语在线观看 | 欧洲熟妇精品视频 | 成人无码视频在线观看网站 | 久久综合香蕉国产蜜臀av | 久久精品女人天堂av免费观看 | 欧美zoozzooz性欧美 | 亚洲精品久久久久avwww潮水 | 免费看男女做好爽好硬视频 | 野狼第一精品社区 | 在线看片无码永久免费视频 | 精品国产一区二区三区四区在线看 | 又紧又大又爽精品一区二区 | 18禁黄网站男男禁片免费观看 | 日韩av无码一区二区三区 | 在教室伦流澡到高潮hnp视频 | 国产精品丝袜黑色高跟鞋 | 麻豆精产国品 | 欧美放荡的少妇 | 亚洲精品久久久久久一区二区 | 国产无遮挡又黄又爽免费视频 | 丰满少妇人妻久久久久久 | 亚洲精品综合一区二区三区在线 | 国产一区二区不卡老阿姨 | 男女性色大片免费网站 | 六月丁香婷婷色狠狠久久 | 亚洲成av人片天堂网无码】 | 一区二区三区乱码在线 | 欧洲 | 久久国语露脸国产精品电影 | 国产精品自产拍在线观看 | 国产黑色丝袜在线播放 | 亚洲精品一区二区三区四区五区 | 福利一区二区三区视频在线观看 | 精品人妻中文字幕有码在线 | 亚洲小说春色综合另类 | 伦伦影院午夜理论片 | 国产成人无码一二三区视频 | a片免费视频在线观看 | 亚洲色偷偷偷综合网 | 亚洲精品欧美二区三区中文字幕 | 国产亚洲视频中文字幕97精品 | 久9re热视频这里只有精品 | 亚洲精品综合五月久久小说 | 国产后入清纯学生妹 | 国产三级精品三级男人的天堂 | 在线a亚洲视频播放在线观看 | 亚洲 a v无 码免 费 成 人 a v | 亚洲 欧美 激情 小说 另类 | 国产在热线精品视频 | 免费人成在线观看网站 | 又粗又大又硬毛片免费看 | 亚洲爆乳精品无码一区二区三区 | 国产av一区二区三区最新精品 | 精品无码一区二区三区的天堂 | 天堂亚洲免费视频 | 欧美人与物videos另类 | 亚洲精品久久久久久一区二区 | 中文字幕无码人妻少妇免费 | 免费无码一区二区三区蜜桃大 | 久久综合狠狠综合久久综合88 | 亚洲一区二区三区在线观看网站 | 久9re热视频这里只有精品 | 高清国产亚洲精品自在久久 | 精品 日韩 国产 欧美 视频 | 美女极度色诱视频国产 | 日本免费一区二区三区最新 | 毛片内射-百度 | 精品亚洲成av人在线观看 | 无码人妻精品一区二区三区下载 | 亚洲国产成人a精品不卡在线 | 国产精华av午夜在线观看 | 国产熟妇高潮叫床视频播放 | 国产黑色丝袜在线播放 | 国产日产欧产精品精品app | 久久综合激激的五月天 | 国产精品a成v人在线播放 | 日本免费一区二区三区最新 | 国产成人无码午夜视频在线观看 | 国产亚洲欧美日韩亚洲中文色 | 国产真实夫妇视频 | 麻豆果冻传媒2021精品传媒一区下载 | 日韩成人一区二区三区在线观看 | 久久久久免费看成人影片 | 精品久久8x国产免费观看 | 中文字幕无线码免费人妻 | 九九综合va免费看 | 亚洲一区二区三区偷拍女厕 | 亚洲国产精品无码一区二区三区 | 自拍偷自拍亚洲精品10p | 国产在线无码精品电影网 | 国产艳妇av在线观看果冻传媒 | 亚拍精品一区二区三区探花 | 狠狠综合久久久久综合网 | 麻豆人妻少妇精品无码专区 | 国产99久久精品一区二区 | 国产偷抇久久精品a片69 | 成在人线av无码免观看麻豆 | 国产av人人夜夜澡人人爽麻豆 | 中文字幕人成乱码熟女app | 国产特级毛片aaaaaaa高清 | 亚洲午夜福利在线观看 | 又黄又爽又色的视频 | a国产一区二区免费入口 | 国产精品久久久 | 亚洲欧美综合区丁香五月小说 | 亚洲小说春色综合另类 | 99视频精品全部免费免费观看 | 中文字幕无码免费久久99 | 亚洲色欲久久久综合网东京热 | 粉嫩少妇内射浓精videos | 亚洲精品中文字幕乱码 | 欧美 丝袜 自拍 制服 另类 | 亚洲精品一区二区三区婷婷月 | 国产av一区二区精品久久凹凸 | 精品亚洲成av人在线观看 | 国产精品办公室沙发 | 丰满人妻一区二区三区免费视频 | 久久婷婷五月综合色国产香蕉 | 麻豆国产97在线 | 欧洲 | 精品国产一区二区三区四区在线看 | 国产三级久久久精品麻豆三级 | 中文字幕人妻无码一夲道 | 久久久久免费看成人影片 | 精品国产aⅴ无码一区二区 | 性做久久久久久久免费看 | 最新国产麻豆aⅴ精品无码 | 国产成人人人97超碰超爽8 | 国产精品无码一区二区三区不卡 | 亚洲高清偷拍一区二区三区 | 日日摸日日碰夜夜爽av | 亚洲日韩av一区二区三区四区 | 国产莉萝无码av在线播放 | 精品水蜜桃久久久久久久 | 男人的天堂av网站 | 成人女人看片免费视频放人 | 狠狠色色综合网站 | 狠狠亚洲超碰狼人久久 | 国产网红无码精品视频 | 思思久久99热只有频精品66 | 中文字幕乱码亚洲无线三区 | 国产亚洲精品久久久久久国模美 | 久久国内精品自在自线 | 无码国产乱人伦偷精品视频 | 欧美日韩视频无码一区二区三 | 97久久国产亚洲精品超碰热 | 一本久久a久久精品vr综合 | 亚洲综合久久一区二区 | www成人国产高清内射 | 理论片87福利理论电影 | 男女下面进入的视频免费午夜 | 国产激情艳情在线看视频 | 高潮毛片无遮挡高清免费 | 免费无码午夜福利片69 | 精品熟女少妇av免费观看 | 97无码免费人妻超级碰碰夜夜 | 国产亚av手机在线观看 | 日韩视频 中文字幕 视频一区 | 天天躁日日躁狠狠躁免费麻豆 | 波多野结衣aⅴ在线 | 婷婷五月综合缴情在线视频 | 婷婷综合久久中文字幕蜜桃三电影 | 在线天堂新版最新版在线8 | a在线亚洲男人的天堂 | 国产精品久久久久9999小说 | 日本在线高清不卡免费播放 | 无码人妻精品一区二区三区不卡 | 欧洲熟妇精品视频 | 精品欧洲av无码一区二区三区 | 久久精品人人做人人综合试看 | 欧美兽交xxxx×视频 | 人人爽人人爽人人片av亚洲 | 狂野欧美性猛xxxx乱大交 | 国产精品久久久久7777 | 国产成人久久精品流白浆 | 国产无套粉嫩白浆在线 | 亚洲成a人一区二区三区 | 99久久久国产精品无码免费 | 亚洲成在人网站无码天堂 | 日本肉体xxxx裸交 | 中国大陆精品视频xxxx | 成人免费视频在线观看 | 欧美性猛交内射兽交老熟妇 | 亚洲精品一区二区三区大桥未久 | 欧美大屁股xxxxhd黑色 | 成人免费视频一区二区 | 日日干夜夜干 | a国产一区二区免费入口 | 欧美国产日韩亚洲中文 | 狠狠综合久久久久综合网 | 亚洲综合无码久久精品综合 | 欧美亚洲日韩国产人成在线播放 | 熟妇人妻无码xxx视频 | 午夜不卡av免费 一本久久a久久精品vr综合 | 中文字幕精品av一区二区五区 | 激情亚洲一区国产精品 | 无遮挡国产高潮视频免费观看 | 又黄又爽又色的视频 | 国产无套内射久久久国产 | 国产乱人无码伦av在线a | 天天摸天天碰天天添 | 亚洲大尺度无码无码专区 | 日韩亚洲欧美中文高清在线 | 在线精品亚洲一区二区 | 在线播放亚洲第一字幕 | 亚洲区欧美区综合区自拍区 | 国产午夜手机精彩视频 | 午夜免费福利小电影 | 成人亚洲精品久久久久软件 | 欧美freesex黑人又粗又大 | av无码电影一区二区三区 | 国产精品亚洲lv粉色 | 欧美人与禽zoz0性伦交 | 国产舌乚八伦偷品w中 | 天天摸天天碰天天添 | 色婷婷综合中文久久一本 | 欧美日韩视频无码一区二区三 | 国产乡下妇女做爰 | 2020久久超碰国产精品最新 | 国内精品人妻无码久久久影院 | 精品亚洲韩国一区二区三区 | 国产亚洲人成在线播放 | 欧美日韩一区二区免费视频 | 丰满少妇人妻久久久久久 | 亚洲综合在线一区二区三区 | 性欧美牲交xxxxx视频 | 国产三级久久久精品麻豆三级 | 99视频精品全部免费免费观看 | 精品亚洲成av人在线观看 | 国产熟妇高潮叫床视频播放 | 精品夜夜澡人妻无码av蜜桃 | 国产精品人妻一区二区三区四 | 日本精品久久久久中文字幕 | 免费人成在线视频无码 | 久久精品国产99精品亚洲 | 一本久道久久综合婷婷五月 | 东京热一精品无码av | 色一情一乱一伦 | 少妇人妻大乳在线视频 | 四虎影视成人永久免费观看视频 | 欧美性生交活xxxxxdddd | 老子影院午夜伦不卡 | 国产两女互慰高潮视频在线观看 | 色情久久久av熟女人妻网站 | 成人亚洲精品久久久久软件 | 国产精品久久久久久无码 | 亚洲热妇无码av在线播放 | 国产亚洲精品久久久久久大师 | 久久久国产精品无码免费专区 | 国产亚洲精品久久久久久大师 | 国产午夜福利100集发布 | 日日摸夜夜摸狠狠摸婷婷 | 国内丰满熟女出轨videos | www国产精品内射老师 | 天天摸天天碰天天添 | 无码人妻精品一区二区三区下载 | 亚洲人成影院在线观看 | 亚洲精品鲁一鲁一区二区三区 | 日本丰满护士爆乳xxxx | 国内丰满熟女出轨videos | 在线成人www免费观看视频 | 亚洲国产日韩a在线播放 | 久久天天躁狠狠躁夜夜免费观看 | 亚洲人成影院在线无码按摩店 | 人妻尝试又大又粗久久 | 蜜桃av抽搐高潮一区二区 | 亚洲一区二区三区无码久久 | 3d动漫精品啪啪一区二区中 | 2019午夜福利不卡片在线 | 少妇性俱乐部纵欲狂欢电影 | 婷婷综合久久中文字幕蜜桃三电影 | 在线看片无码永久免费视频 | 国产片av国语在线观看 | 久久午夜无码鲁丝片秋霞 | 麻豆人妻少妇精品无码专区 | 人妻熟女一区 | 国内精品久久久久久中文字幕 | 六月丁香婷婷色狠狠久久 | 亚洲国产精品毛片av不卡在线 | 内射欧美老妇wbb | 18精品久久久无码午夜福利 | 88国产精品欧美一区二区三区 | 国产一区二区不卡老阿姨 | 亚洲人成人无码网www国产 | 成 人 免费观看网站 | 高潮毛片无遮挡高清免费视频 | 奇米综合四色77777久久 东京无码熟妇人妻av在线网址 | 亚洲男人av天堂午夜在 | 人人爽人人澡人人人妻 | 欧美日韩一区二区免费视频 | 亚洲精品久久久久avwww潮水 | 99久久久无码国产精品免费 | 亚洲国产日韩a在线播放 | 丰满肥臀大屁股熟妇激情视频 | 国产人妻精品一区二区三区 | 人人爽人人澡人人人妻 | 亚洲精品成人福利网站 | 色 综合 欧美 亚洲 国产 | 久久综合给久久狠狠97色 | 国产一精品一av一免费 | 狠狠躁日日躁夜夜躁2020 | 天天爽夜夜爽夜夜爽 | 国产97色在线 | 免 | 亚洲综合色区中文字幕 | 鲁一鲁av2019在线 | 久久99精品国产麻豆 | 精品一区二区三区波多野结衣 | 人妻无码久久精品人妻 | 免费人成在线观看网站 | 国产精品久久久久9999小说 | 久久综合激激的五月天 | 亚洲精品国产a久久久久久 | 欧美精品一区二区精品久久 | 国内精品人妻无码久久久影院蜜桃 | 亚洲中文字幕无码一久久区 | 久久久久亚洲精品中文字幕 | 午夜成人1000部免费视频 | 全球成人中文在线 | 亚洲爆乳精品无码一区二区三区 | 精品国产精品久久一区免费式 | 兔费看少妇性l交大片免费 | 国产精品视频免费播放 | 国产精品99爱免费视频 | 国产精品a成v人在线播放 | 日韩少妇内射免费播放 | 永久免费观看美女裸体的网站 | av人摸人人人澡人人超碰下载 | 少妇人妻av毛片在线看 | 强辱丰满人妻hd中文字幕 | 性色av无码免费一区二区三区 | 久久99国产综合精品 | 人人爽人人爽人人片av亚洲 | 欧美国产日产一区二区 | 一区二区三区乱码在线 | 欧洲 | 免费人成在线观看网站 | 荫蒂被男人添的好舒服爽免费视频 | 亚洲国产精品久久久久久 | 免费人成在线视频无码 | 国产熟妇高潮叫床视频播放 | 中文字幕精品av一区二区五区 | 国产明星裸体无码xxxx视频 | 日本爽爽爽爽爽爽在线观看免 | 无码av最新清无码专区吞精 | 国产两女互慰高潮视频在线观看 | 亚洲精品一区二区三区大桥未久 | 欧美怡红院免费全部视频 | 四虎影视成人永久免费观看视频 | 色综合久久久无码中文字幕 | 国产凸凹视频一区二区 | 国产成人精品视频ⅴa片软件竹菊 | 国产人妻人伦精品 | 日日橹狠狠爱欧美视频 | 日韩在线不卡免费视频一区 | 国语自产偷拍精品视频偷 | 欧美日韩一区二区免费视频 | 日本精品高清一区二区 | 久久午夜无码鲁丝片午夜精品 | 欧美人与善在线com | 亚洲综合另类小说色区 | 国产成人亚洲综合无码 | a片免费视频在线观看 | 大屁股大乳丰满人妻 | 人妻人人添人妻人人爱 | 日产精品高潮呻吟av久久 | 最近免费中文字幕中文高清百度 | 丰满人妻一区二区三区免费视频 | 欧美日韩综合一区二区三区 | 一本久道高清无码视频 | 伊人久久大香线焦av综合影院 | 亚洲熟悉妇女xxx妇女av | 欧美丰满少妇xxxx性 | 久久精品国产99精品亚洲 | 免费无码午夜福利片69 | 欧美zoozzooz性欧美 | 色综合久久中文娱乐网 | 日日摸天天摸爽爽狠狠97 | 国产人妻精品一区二区三区不卡 | 亚洲精品国产a久久久久久 | 国产成人亚洲综合无码 | 99精品无人区乱码1区2区3区 | 久久亚洲中文字幕无码 | 色一情一乱一伦一区二区三欧美 | 午夜男女很黄的视频 | 午夜福利一区二区三区在线观看 | 日韩无码专区 | 国产亚洲精品精品国产亚洲综合 | 国内精品久久毛片一区二区 | 男人的天堂av网站 | 狠狠色噜噜狠狠狠7777奇米 | 亚洲人亚洲人成电影网站色 | 亚洲无人区午夜福利码高清完整版 | 无码乱肉视频免费大全合集 | 人妻少妇被猛烈进入中文字幕 | 人人妻人人澡人人爽人人精品 | 欧美猛少妇色xxxxx | 天天摸天天碰天天添 | 麻豆精品国产精华精华液好用吗 | 欧美精品无码一区二区三区 | 国产成人无码一二三区视频 | 免费观看的无遮挡av | 高清国产亚洲精品自在久久 | 麻豆av传媒蜜桃天美传媒 | 中文字幕 亚洲精品 第1页 | 蜜桃无码一区二区三区 | 亚洲毛片av日韩av无码 | 国产成人精品久久亚洲高清不卡 | 熟女少妇人妻中文字幕 | 夫妻免费无码v看片 | 国产精品99爱免费视频 | 三上悠亚人妻中文字幕在线 | 亚洲区小说区激情区图片区 | 免费视频欧美无人区码 | 色婷婷综合激情综在线播放 | 性色av无码免费一区二区三区 | 人人妻人人藻人人爽欧美一区 | 天天拍夜夜添久久精品大 | 国产精品内射视频免费 | 色五月丁香五月综合五月 | 3d动漫精品啪啪一区二区中 | 中文无码成人免费视频在线观看 | 亚洲爆乳无码专区 | 青青青爽视频在线观看 | 综合人妻久久一区二区精品 | 亚洲国产欧美日韩精品一区二区三区 | 老子影院午夜伦不卡 | 国产成人精品久久亚洲高清不卡 | 国产免费久久久久久无码 | 性生交大片免费看女人按摩摩 | 欧美日韩一区二区三区自拍 | 久久午夜无码鲁丝片秋霞 | 精品无码av一区二区三区 | 色偷偷av老熟女 久久精品人妻少妇一区二区三区 | 欧美日韩一区二区三区自拍 | 成人女人看片免费视频放人 | 草草网站影院白丝内射 | 亚洲 另类 在线 欧美 制服 | 亚洲精品综合一区二区三区在线 | 东京无码熟妇人妻av在线网址 | 精品成人av一区二区三区 | 国产特级毛片aaaaaa高潮流水 | 亚洲高清偷拍一区二区三区 | 97久久国产亚洲精品超碰热 | 亚洲无人区午夜福利码高清完整版 | 黑人粗大猛烈进出高潮视频 | 高潮毛片无遮挡高清免费视频 | 天海翼激烈高潮到腰振不止 | 无码一区二区三区在线观看 | 婷婷五月综合激情中文字幕 | 亚洲精品国产精品乱码不卡 | 色婷婷av一区二区三区之红樱桃 | 久久99精品国产麻豆蜜芽 | 国产午夜亚洲精品不卡下载 | 99riav国产精品视频 | 中文字幕乱妇无码av在线 | 国产午夜福利亚洲第一 | 免费无码的av片在线观看 | aa片在线观看视频在线播放 | 色欲av亚洲一区无码少妇 | 中文字幕无码av波多野吉衣 | 国产舌乚八伦偷品w中 | 日韩人妻无码中文字幕视频 | 乱码av麻豆丝袜熟女系列 | 精品国产麻豆免费人成网站 | 欧美日韩一区二区免费视频 | 99er热精品视频 | 在线亚洲高清揄拍自拍一品区 | 欧美黑人乱大交 | 麻豆果冻传媒2021精品传媒一区下载 | 亚洲日本一区二区三区在线 | 国产亚洲美女精品久久久2020 | 国产午夜无码视频在线观看 | 成熟女人特级毛片www免费 | 久久久久久久人妻无码中文字幕爆 | 日本一区二区三区免费播放 | 国产热a欧美热a在线视频 | 亚洲精品一区二区三区在线 | 久热国产vs视频在线观看 | 欧美黑人巨大xxxxx | 麻豆果冻传媒2021精品传媒一区下载 | 特大黑人娇小亚洲女 | 美女毛片一区二区三区四区 | 2020久久香蕉国产线看观看 | 欧美老妇交乱视频在线观看 | 久久人妻内射无码一区三区 | 亚洲の无码国产の无码影院 | 日日摸天天摸爽爽狠狠97 | 国产成人一区二区三区别 | 欧美性猛交xxxx富婆 | 99久久精品无码一区二区毛片 | 国产成人精品视频ⅴa片软件竹菊 | 午夜免费福利小电影 | а√天堂www在线天堂小说 | 日本精品高清一区二区 | 东京热无码av男人的天堂 | 少妇人妻大乳在线视频 | 国产性生大片免费观看性 | 亚洲精品成a人在线观看 | 搡女人真爽免费视频大全 | 未满小14洗澡无码视频网站 | 国产精品久久久久久亚洲毛片 | 欧美精品免费观看二区 | 永久黄网站色视频免费直播 | 黄网在线观看免费网站 | 少妇高潮喷潮久久久影院 | 美女毛片一区二区三区四区 | 国产一区二区三区四区五区加勒比 | 欧美日韩在线亚洲综合国产人 | 国产真实伦对白全集 | aⅴ在线视频男人的天堂 | 久久99精品久久久久久动态图 | 亚洲经典千人经典日产 | 久久人人爽人人爽人人片ⅴ | 少女韩国电视剧在线观看完整 | 丰满诱人的人妻3 | 中文字幕无码免费久久9一区9 | 熟妇女人妻丰满少妇中文字幕 | 国产熟妇另类久久久久 | 美女极度色诱视频国产 | 天天综合网天天综合色 | 精品一区二区三区无码免费视频 | 亚洲经典千人经典日产 | 亚洲熟熟妇xxxx | 久久精品99久久香蕉国产色戒 | 国产人成高清在线视频99最全资源 | 成人免费无码大片a毛片 | 免费无码的av片在线观看 | 亚洲熟悉妇女xxx妇女av | 精品水蜜桃久久久久久久 | 人妻插b视频一区二区三区 | 爆乳一区二区三区无码 | 国产美女精品一区二区三区 | 精品人人妻人人澡人人爽人人 | 国产在线一区二区三区四区五区 | 双乳奶水饱满少妇呻吟 | 亚洲男人av天堂午夜在 | 日本一卡2卡3卡四卡精品网站 | 亚洲の无码国产の无码影院 | 中文久久乱码一区二区 | 亚洲日韩av一区二区三区中文 | 国产午夜无码精品免费看 | 在线播放免费人成毛片乱码 | 欧美成人高清在线播放 | 亚洲精品一区二区三区在线观看 | 国产特级毛片aaaaaa高潮流水 | 国产美女精品一区二区三区 | 精品成人av一区二区三区 | 精品少妇爆乳无码av无码专区 | 欧美色就是色 | 九九综合va免费看 | 中文字幕无码乱人伦 | 亚洲日韩精品欧美一区二区 | 亚洲成av人片在线观看无码不卡 | 九九在线中文字幕无码 | 国精产品一品二品国精品69xx | 又大又黄又粗又爽的免费视频 | 欧美色就是色 | 成人av无码一区二区三区 | 两性色午夜免费视频 | 国产人妻大战黑人第1集 | 2020久久超碰国产精品最新 | 国产精品久久久久久亚洲影视内衣 | 日本在线高清不卡免费播放 | 国产性生交xxxxx无码 | 天天做天天爱天天爽综合网 | www成人国产高清内射 | 图片区 小说区 区 亚洲五月 | 色情久久久av熟女人妻网站 | 久久国产精品萌白酱免费 | 又粗又大又硬毛片免费看 | 毛片内射-百度 | 麻豆精品国产精华精华液好用吗 | 特级做a爰片毛片免费69 | 激情五月综合色婷婷一区二区 | 综合激情五月综合激情五月激情1 | 国产麻豆精品精东影业av网站 | 亚洲精品一区二区三区在线观看 | 免费无码的av片在线观看 | 国产99久久精品一区二区 | 在线欧美精品一区二区三区 | 国产乱码精品一品二品 | 波多野结衣高清一区二区三区 | 国内精品一区二区三区不卡 | 国产成人综合在线女婷五月99播放 | 无码帝国www无码专区色综合 | 丰满人妻翻云覆雨呻吟视频 | 我要看www免费看插插视频 | 国产热a欧美热a在线视频 | 一本色道婷婷久久欧美 | 日本在线高清不卡免费播放 | 又紧又大又爽精品一区二区 | 色婷婷综合中文久久一本 | 久久久久成人精品免费播放动漫 | 国产精品办公室沙发 | 国产欧美精品一区二区三区 | 亚洲色大成网站www | 国精产品一品二品国精品69xx | 国产精品无码mv在线观看 | 国产成人综合在线女婷五月99播放 | 好爽又高潮了毛片免费下载 | 亚洲成a人片在线观看无码 | 领导边摸边吃奶边做爽在线观看 | 国产69精品久久久久app下载 | 国产精品99爱免费视频 | 亚洲精品国产品国语在线观看 | 欧美丰满熟妇xxxx | 日韩av激情在线观看 | 国产成人无码专区 | 无码国内精品人妻少妇 | 亚洲欧美中文字幕5发布 | 精品国产精品久久一区免费式 | 超碰97人人射妻 | 亚洲欧美日韩成人高清在线一区 | 欧美刺激性大交 | 窝窝午夜理论片影院 | 欧美阿v高清资源不卡在线播放 | 51国偷自产一区二区三区 | 少妇无套内谢久久久久 | 国产色在线 | 国产 | 国产日产欧产精品精品app | 国产精品人妻一区二区三区四 | 国产熟妇高潮叫床视频播放 | 精品久久久中文字幕人妻 | 国产偷自视频区视频 | 国产精品99爱免费视频 | 在线视频网站www色 | 精品无码av一区二区三区 | 丝袜人妻一区二区三区 | 久久人妻内射无码一区三区 | 午夜精品一区二区三区的区别 | 97无码免费人妻超级碰碰夜夜 | 日韩精品久久久肉伦网站 | 成人精品一区二区三区中文字幕 | 色 综合 欧美 亚洲 国产 | 久久人妻内射无码一区三区 | 国产福利视频一区二区 | 亚洲男人av香蕉爽爽爽爽 | 熟妇人妻无乱码中文字幕 | 麻豆蜜桃av蜜臀av色欲av | 永久免费观看美女裸体的网站 | 欧美性猛交xxxx富婆 | 天堂亚洲免费视频 | 俄罗斯老熟妇色xxxx | 妺妺窝人体色www婷婷 | 亚洲理论电影在线观看 | 性啪啪chinese东北女人 | 国产精品美女久久久网av | 免费无码av一区二区 | 装睡被陌生人摸出水好爽 | 亚洲精品国偷拍自产在线麻豆 | 久久久久99精品成人片 | 欧美刺激性大交 | 性色欲情网站iwww九文堂 | 一二三四社区在线中文视频 | 欧美日韩视频无码一区二区三 | 99久久久国产精品无码免费 | 国产在热线精品视频 | 国产精品高潮呻吟av久久4虎 | 国产精品久久久久久久影院 | 狠狠色色综合网站 | 国产精品美女久久久久av爽李琼 | 国产精品欧美成人 | 午夜精品久久久久久久 | 国产成人综合美国十次 | 国产午夜亚洲精品不卡 | 少妇性l交大片欧洲热妇乱xxx | 国产精品手机免费 | 又大又硬又爽免费视频 | yw尤物av无码国产在线观看 | 少妇的肉体aa片免费 | 在线播放亚洲第一字幕 | aa片在线观看视频在线播放 | 性欧美疯狂xxxxbbbb | 少妇激情av一区二区 | 少妇厨房愉情理9仑片视频 | 日本xxxx色视频在线观看免费 | 色婷婷久久一区二区三区麻豆 | 领导边摸边吃奶边做爽在线观看 | 色一情一乱一伦一区二区三欧美 | 老熟女重囗味hdxx69 | 国产欧美精品一区二区三区 | 捆绑白丝粉色jk震动捧喷白浆 | 丰满妇女强制高潮18xxxx | 少妇厨房愉情理9仑片视频 | 99久久久国产精品无码免费 | 丰满少妇高潮惨叫视频 | 蜜桃av蜜臀av色欲av麻 999久久久国产精品消防器材 | 欧美阿v高清资源不卡在线播放 | 色诱久久久久综合网ywww | 精品人人妻人人澡人人爽人人 | 国产人妻大战黑人第1集 | 国产美女精品一区二区三区 | 97久久国产亚洲精品超碰热 | 成人av无码一区二区三区 | 俺去俺来也www色官网 | 国产成人精品三级麻豆 | 国产在线aaa片一区二区99 | 欧美肥老太牲交大战 | 国产精品久久久午夜夜伦鲁鲁 | 久久亚洲中文字幕无码 | 无套内谢的新婚少妇国语播放 | 国模大胆一区二区三区 | 国内精品九九久久久精品 | 亚洲人交乣女bbw | 午夜无码人妻av大片色欲 | 欧美阿v高清资源不卡在线播放 | 国产激情精品一区二区三区 | 红桃av一区二区三区在线无码av | 欧美精品国产综合久久 | 国产精品理论片在线观看 | 黑森林福利视频导航 | 国产凸凹视频一区二区 | 国产69精品久久久久app下载 | 国产精品无套呻吟在线 | 亚洲欧洲中文日韩av乱码 | 国产成人精品一区二区在线小狼 | 国产精品爱久久久久久久 | 狠狠综合久久久久综合网 | 亚洲国产精品久久人人爱 | 啦啦啦www在线观看免费视频 | 四十如虎的丰满熟妇啪啪 | 亚洲人成网站免费播放 | 暴力强奷在线播放无码 | 丰满少妇高潮惨叫视频 | 无码人妻精品一区二区三区下载 | 少妇被粗大的猛进出69影院 | 熟女少妇在线视频播放 | 欧美变态另类xxxx | 欧美第一黄网免费网站 | 牲欲强的熟妇农村老妇女视频 | 色欲久久久天天天综合网精品 | 一本久道久久综合婷婷五月 | 亚洲高清偷拍一区二区三区 | 曰韩少妇内射免费播放 | 无码人中文字幕 | 国产精品人人妻人人爽 | 国产成人av免费观看 | 99国产欧美久久久精品 | 青青青爽视频在线观看 | 无码人妻出轨黑人中文字幕 | 久久久无码中文字幕久... | 婷婷五月综合激情中文字幕 | 97夜夜澡人人双人人人喊 | 小sao货水好多真紧h无码视频 | 男人扒开女人内裤强吻桶进去 | 色妞www精品免费视频 | 欧美人与禽猛交狂配 | 欧美放荡的少妇 | 国产激情无码一区二区app | 丝袜美腿亚洲一区二区 | 人妻少妇精品无码专区动漫 | 亚洲欧洲中文日韩av乱码 | 一个人看的www免费视频在线观看 | 强奷人妻日本中文字幕 | 久久精品一区二区三区四区 | 国产成人无码av片在线观看不卡 | 国产日产欧产精品精品app | 国产农村妇女高潮大叫 | 九九在线中文字幕无码 | 成年美女黄网站色大免费视频 | 国产成人精品久久亚洲高清不卡 | 免费人成网站视频在线观看 | 久久99久久99精品中文字幕 | 国产无遮挡又黄又爽免费视频 | 成 人影片 免费观看 | 欧美亚洲国产一区二区三区 | 欧美日韩一区二区三区自拍 | 中文久久乱码一区二区 | 国产亚洲精品久久久久久国模美 | 欧美人与物videos另类 | 兔费看少妇性l交大片免费 | 青草青草久热国产精品 | 任你躁国产自任一区二区三区 | 国产内射老熟女aaaa | 51国偷自产一区二区三区 | 亚洲成av人片在线观看无码不卡 | 色综合久久久无码中文字幕 | 精品无码一区二区三区的天堂 | 奇米影视888欧美在线观看 | 亚洲s码欧洲m码国产av | 无码av岛国片在线播放 | 亚洲精品国偷拍自产在线麻豆 | av在线亚洲欧洲日产一区二区 | 377p欧洲日本亚洲大胆 | 日日天干夜夜狠狠爱 | 国产亚av手机在线观看 | 亚洲熟妇自偷自拍另类 | 色婷婷久久一区二区三区麻豆 | 97人妻精品一区二区三区 | 日日夜夜撸啊撸 | 久久国产精品萌白酱免费 | 福利一区二区三区视频在线观看 | 亚洲欧洲中文日韩av乱码 | 色欲人妻aaaaaaa无码 | 亚洲人成影院在线无码按摩店 | 99riav国产精品视频 | 好男人社区资源 | 日日碰狠狠躁久久躁蜜桃 | 国产人妻精品一区二区三区 | 久久久久成人精品免费播放动漫 | 无码国产乱人伦偷精品视频 | 中文字幕无线码免费人妻 | 国产精品美女久久久 | 在线看片无码永久免费视频 | 少妇性俱乐部纵欲狂欢电影 | 国产偷抇久久精品a片69 | 亚洲精品中文字幕乱码 | 国产内射爽爽大片视频社区在线 | 中文字幕无码免费久久9一区9 | 久久久精品456亚洲影院 | 男人和女人高潮免费网站 | 国产三级久久久精品麻豆三级 | 日产精品高潮呻吟av久久 | 国产精品亚洲а∨无码播放麻豆 | 久久五月精品中文字幕 | 国产肉丝袜在线观看 | 久久久国产一区二区三区 | 久久久精品456亚洲影院 | 精品国产福利一区二区 | 日韩无码专区 | 美女扒开屁股让男人桶 | 67194成是人免费无码 | 亚洲の无码国产の无码影院 | 无码av中文字幕免费放 | 亚洲色无码一区二区三区 | 永久免费观看国产裸体美女 | 国产9 9在线 | 中文 | 成人欧美一区二区三区黑人 | 麻豆国产人妻欲求不满 | 精品无码一区二区三区爱欲 | 色五月丁香五月综合五月 | 国内老熟妇对白xxxxhd | 扒开双腿疯狂进出爽爽爽视频 | 国产亚洲视频中文字幕97精品 | 影音先锋中文字幕无码 | 国产舌乚八伦偷品w中 | 亚洲人成网站在线播放942 | 九月婷婷人人澡人人添人人爽 | 国内精品人妻无码久久久影院蜜桃 | 性欧美熟妇videofreesex | 亚洲一区二区三区播放 | 内射巨臀欧美在线视频 | 久久国产精品偷任你爽任你 | 无码人妻精品一区二区三区下载 | 天海翼激烈高潮到腰振不止 | 无码精品国产va在线观看dvd | 老熟女重囗味hdxx69 | 色窝窝无码一区二区三区色欲 | 无码人妻精品一区二区三区不卡 | 精品一区二区三区无码免费视频 | 国产9 9在线 | 中文 | 任你躁国产自任一区二区三区 | 夫妻免费无码v看片 | 99久久人妻精品免费一区 | 啦啦啦www在线观看免费视频 | 中文无码成人免费视频在线观看 | 成人精品天堂一区二区三区 | 欧美精品无码一区二区三区 | 我要看www免费看插插视频 | 国产午夜视频在线观看 | 澳门永久av免费网站 | 久久国产精品二国产精品 | 丰满肥臀大屁股熟妇激情视频 | 国产精品99久久精品爆乳 | 欧美兽交xxxx×视频 | 蜜桃视频插满18在线观看 | 俄罗斯老熟妇色xxxx | 99久久精品国产一区二区蜜芽 | 久久综合网欧美色妞网 | 97久久国产亚洲精品超碰热 | 麻豆果冻传媒2021精品传媒一区下载 | 国产乱人伦av在线无码 | 久久婷婷五月综合色国产香蕉 | 亚洲熟妇色xxxxx欧美老妇 | 成人av无码一区二区三区 | 国产成人精品必看 | 波多野结衣av在线观看 | 国产香蕉尹人视频在线 | 呦交小u女精品视频 | 国产亚洲精品精品国产亚洲综合 | 亚洲 高清 成人 动漫 | 亚洲狠狠婷婷综合久久 | 人人妻人人澡人人爽人人精品 | 国产在线精品一区二区高清不卡 | 97无码免费人妻超级碰碰夜夜 | 成在人线av无码免费 | 一区二区三区乱码在线 | 欧洲 | 久久国产自偷自偷免费一区调 | 亚洲无人区午夜福利码高清完整版 | 午夜成人1000部免费视频 | 亚洲中文字幕成人无码 | 国产黄在线观看免费观看不卡 | 欧美黑人乱大交 | 综合人妻久久一区二区精品 | 亚洲精品国产精品乱码不卡 | 国産精品久久久久久久 | 麻豆精品国产精华精华液好用吗 | 欧美乱妇无乱码大黄a片 | 丰满岳乱妇在线观看中字无码 | 久久午夜无码鲁丝片秋霞 | 精品国产福利一区二区 | 欧美放荡的少妇 | 成人综合网亚洲伊人 | 国产精品无码成人午夜电影 | av在线亚洲欧洲日产一区二区 | 亚洲理论电影在线观看 | 成人三级无码视频在线观看 | 中文亚洲成a人片在线观看 | av人摸人人人澡人人超碰下载 | 日韩精品无码一本二本三本色 | 一个人看的视频www在线 | www国产亚洲精品久久久日本 | 国产色精品久久人妻 | 中文字幕 亚洲精品 第1页 | 少妇性l交大片欧洲热妇乱xxx | 亚洲日本va午夜在线电影 | 漂亮人妻洗澡被公强 日日躁 | 亚洲国产av精品一区二区蜜芽 | 丰满少妇高潮惨叫视频 | ass日本丰满熟妇pics | 波多野结衣av在线观看 | 男女性色大片免费网站 | 极品嫩模高潮叫床 | 亚洲国产欧美国产综合一区 | 国产偷国产偷精品高清尤物 | 亚洲成在人网站无码天堂 | 中文无码成人免费视频在线观看 | 国产精品嫩草久久久久 | 亚洲精品国产精品乱码不卡 | 亚洲另类伦春色综合小说 | 国产成人无码区免费内射一片色欲 | 最新版天堂资源中文官网 | 俺去俺来也www色官网 | 欧美国产日韩久久mv | 国产精品无码永久免费888 | 亚洲国精产品一二二线 | 毛片内射-百度 | 亚洲s码欧洲m码国产av | 亚洲最大成人网站 | 国产偷自视频区视频 | 亚洲中文字幕无码一久久区 | 精品夜夜澡人妻无码av蜜桃 | 婷婷六月久久综合丁香 | 国产亚洲精品久久久ai换 | 国产在热线精品视频 | 国产成人无码区免费内射一片色欲 | 国产av一区二区精品久久凹凸 | 一本无码人妻在中文字幕免费 | 蜜桃无码一区二区三区 | 鲁一鲁av2019在线 | 久9re热视频这里只有精品 | 少妇无套内谢久久久久 | www成人国产高清内射 | 久久久国产一区二区三区 | 日本乱偷人妻中文字幕 | 97色伦图片97综合影院 | 2019nv天堂香蕉在线观看 | 无码人妻丰满熟妇区五十路百度 | 美女扒开屁股让男人桶 | 亚洲国产成人av在线观看 | 成人无码精品一区二区三区 | 亚洲 另类 在线 欧美 制服 | 综合激情五月综合激情五月激情1 | 日本爽爽爽爽爽爽在线观看免 | 精品偷拍一区二区三区在线看 | 国产精品毛片一区二区 | 欧美阿v高清资源不卡在线播放 | 亚无码乱人伦一区二区 | 亚洲欧洲日本无在线码 | 性生交大片免费看女人按摩摩 | 亚洲中文字幕在线无码一区二区 | 国产精品久久久久7777 | 久久久亚洲欧洲日产国码αv | 亚洲日韩中文字幕在线播放 | 67194成是人免费无码 | 捆绑白丝粉色jk震动捧喷白浆 | 久久国产自偷自偷免费一区调 | 国产亚洲欧美日韩亚洲中文色 | 亚洲一区二区三区含羞草 | 欧美高清在线精品一区 | 日韩在线不卡免费视频一区 | 免费观看的无遮挡av | 国产在线精品一区二区高清不卡 | 国产网红无码精品视频 | 小泽玛莉亚一区二区视频在线 | 性做久久久久久久久 | 青青久在线视频免费观看 | 人人超人人超碰超国产 | 99国产精品白浆在线观看免费 | 无码吃奶揉捏奶头高潮视频 | 欧美精品一区二区精品久久 | 麻豆成人精品国产免费 | 亚洲日韩av一区二区三区四区 | 激情综合激情五月俺也去 | 双乳奶水饱满少妇呻吟 | 日日干夜夜干 | 久久久www成人免费毛片 | 国产一区二区三区精品视频 | 无码av中文字幕免费放 | 亚洲精品一区二区三区婷婷月 | 国产人成高清在线视频99最全资源 | 国产福利视频一区二区 | 国产香蕉尹人视频在线 | 亚洲国产精品久久久天堂 | 中文字幕 人妻熟女 | 欧美猛少妇色xxxxx | 天堂亚洲免费视频 | 国产片av国语在线观看 | 亚洲va欧美va天堂v国产综合 | 国产成人精品三级麻豆 | 亚洲国产成人a精品不卡在线 | 亚洲午夜久久久影院 | 精品人人妻人人澡人人爽人人 | 国产香蕉尹人视频在线 | 巨爆乳无码视频在线观看 | 1000部啪啪未满十八勿入下载 | 国产一区二区不卡老阿姨 | 色五月丁香五月综合五月 | 亚洲精品国产品国语在线观看 | 中文无码伦av中文字幕 | 无码人妻丰满熟妇区毛片18 | 久久zyz资源站无码中文动漫 | 亚洲欧美日韩国产精品一区二区 | 欧美日韩一区二区免费视频 | 波多野结衣aⅴ在线 | 午夜性刺激在线视频免费 | 亚洲国产精品一区二区美利坚 | 激情人妻另类人妻伦 | 美女黄网站人色视频免费国产 | 国产亲子乱弄免费视频 | 色老头在线一区二区三区 | 小sao货水好多真紧h无码视频 | 日本一区二区三区免费高清 | 图片小说视频一区二区 | 日韩av无码一区二区三区 | 色婷婷欧美在线播放内射 | 亚洲国产欧美日韩精品一区二区三区 | 国产精品99久久精品爆乳 | 日韩精品久久久肉伦网站 | 日日躁夜夜躁狠狠躁 | 色综合天天综合狠狠爱 | 成在人线av无码免观看麻豆 | 婷婷五月综合激情中文字幕 | 亚洲区欧美区综合区自拍区 | yw尤物av无码国产在线观看 | 精品水蜜桃久久久久久久 | 久久国产精品偷任你爽任你 | 亚洲第一网站男人都懂 | 国产乱人伦app精品久久 国产在线无码精品电影网 国产国产精品人在线视 | 99视频精品全部免费免费观看 | 俄罗斯老熟妇色xxxx | 免费观看激色视频网站 | 婷婷丁香六月激情综合啪 | 欧美肥老太牲交大战 | 性史性农村dvd毛片 | 国产精品怡红院永久免费 | 在线欧美精品一区二区三区 | 最新国产乱人伦偷精品免费网站 | 日本在线高清不卡免费播放 | 欧美日韩色另类综合 | 老熟妇仑乱视频一区二区 | 国产成人精品无码播放 | 欧美野外疯狂做受xxxx高潮 | 老司机亚洲精品影院无码 | 成年美女黄网站色大免费全看 | 国产农村乱对白刺激视频 | 中文字幕亚洲情99在线 | 国产成人精品三级麻豆 | 欧美人与牲动交xxxx | 精品人人妻人人澡人人爽人人 | 免费看男女做好爽好硬视频 | 欧美一区二区三区视频在线观看 | 漂亮人妻洗澡被公强 日日躁 | 亚洲 日韩 欧美 成人 在线观看 | 天天av天天av天天透 | 无码人妻丰满熟妇区五十路百度 | 天天躁日日躁狠狠躁免费麻豆 | 欧美熟妇另类久久久久久多毛 | 特级做a爰片毛片免费69 | 鲁鲁鲁爽爽爽在线视频观看 | 亚洲а∨天堂久久精品2021 | 亚洲中文字幕av在天堂 | 丰满少妇女裸体bbw | 国内揄拍国内精品人妻 | 欧洲vodafone精品性 | 婷婷色婷婷开心五月四房播播 | yw尤物av无码国产在线观看 | 色偷偷人人澡人人爽人人模 | 亚洲精品一区三区三区在线观看 | 国产在热线精品视频 | 国产一区二区三区精品视频 | 亚洲午夜福利在线观看 | 欧美人妻一区二区三区 | 亚洲欧洲日本无在线码 | 欧美激情内射喷水高潮 | 高清国产亚洲精品自在久久 | 无遮挡国产高潮视频免费观看 | 超碰97人人射妻 | 亚洲色大成网站www | 成人免费视频视频在线观看 免费 | 久久久久99精品国产片 | 国产人妻大战黑人第1集 | 欧美激情综合亚洲一二区 | 亚洲综合色区中文字幕 | 亚洲日韩乱码中文无码蜜桃臀网站 | 亚洲欧美国产精品久久 | 综合激情五月综合激情五月激情1 | 国产在线aaa片一区二区99 | 日本乱人伦片中文三区 | 丝袜 中出 制服 人妻 美腿 | 曰韩少妇内射免费播放 | 最新国产乱人伦偷精品免费网站 | 免费国产黄网站在线观看 | 亚洲欧洲日本综合aⅴ在线 | 精品一区二区不卡无码av | 中国女人内谢69xxxx | 四虎影视成人永久免费观看视频 | 天天做天天爱天天爽综合网 | 亚洲熟妇自偷自拍另类 | 国产成人无码av片在线观看不卡 | 中文毛片无遮挡高清免费 | 欧美三级a做爰在线观看 | 亚洲精品一区二区三区婷婷月 | 黑人粗大猛烈进出高潮视频 | 欧美人妻一区二区三区 | 无码成人精品区在线观看 | 激情综合激情五月俺也去 | 丰满人妻翻云覆雨呻吟视频 | 国内精品久久久久久中文字幕 | 亚洲精品综合五月久久小说 | 亚洲精品欧美二区三区中文字幕 | 一个人看的www免费视频在线观看 | 麻豆国产丝袜白领秘书在线观看 | av在线亚洲欧洲日产一区二区 | 欧美肥老太牲交大战 | 国产亚洲视频中文字幕97精品 | 丰满人妻翻云覆雨呻吟视频 | 午夜成人1000部免费视频 | 亚洲中文字幕va福利 | 亚洲精品成人福利网站 | 国产精品办公室沙发 | 久久国产精品偷任你爽任你 | 思思久久99热只有频精品66 | a国产一区二区免费入口 | 乱码午夜-极国产极内射 | 四虎影视成人永久免费观看视频 | 亚洲熟妇色xxxxx欧美老妇 | 成人av无码一区二区三区 | 婷婷五月综合激情中文字幕 | 色综合久久久无码网中文 | 中文字幕人成乱码熟女app | 国语自产偷拍精品视频偷 | 在教室伦流澡到高潮hnp视频 | 蜜桃视频韩日免费播放 | 日日天干夜夜狠狠爱 | 少妇一晚三次一区二区三区 | 成人精品一区二区三区中文字幕 | 久久精品一区二区三区四区 | 精品久久久久香蕉网 | 在线精品国产一区二区三区 | 欧美老妇交乱视频在线观看 | 国产美女精品一区二区三区 | 免费中文字幕日韩欧美 | 性欧美牲交在线视频 | 牲欲强的熟妇农村老妇女 | 性史性农村dvd毛片 | 鲁大师影院在线观看 | 国产av无码专区亚洲a∨毛片 | 国产人妻精品一区二区三区 | 欧美性猛交内射兽交老熟妇 | 18禁黄网站男男禁片免费观看 | 久久久久se色偷偷亚洲精品av | 久久99精品国产麻豆 | 色综合久久久无码中文字幕 | 少妇邻居内射在线 | 国产精品沙发午睡系列 | 精品久久综合1区2区3区激情 | 日本爽爽爽爽爽爽在线观看免 | 色综合久久中文娱乐网 | 亚洲精品美女久久久久久久 | 无码午夜成人1000部免费视频 | 亚洲日韩av片在线观看 | 亚洲中文字幕无码中字 | 97se亚洲精品一区 | 国产精品久久久av久久久 | 欧美 丝袜 自拍 制服 另类 | 一区二区三区高清视频一 | 最近免费中文字幕中文高清百度 | 无码人妻少妇伦在线电影 | 丰满岳乱妇在线观看中字无码 | 图片小说视频一区二区 | 在线观看国产一区二区三区 | 亚洲国产一区二区三区在线观看 | 国产精品欧美成人 | 国产麻豆精品一区二区三区v视界 | 俄罗斯老熟妇色xxxx | 夜精品a片一区二区三区无码白浆 | 久久久无码中文字幕久... | 黑人大群体交免费视频 | 377p欧洲日本亚洲大胆 | 中文字幕乱码亚洲无线三区 | 少妇厨房愉情理9仑片视频 | 国产成人精品久久亚洲高清不卡 | 日韩人妻少妇一区二区三区 | 亚洲天堂2017无码 | 国产精品高潮呻吟av久久4虎 | 天堂久久天堂av色综合 | 日本一区二区三区免费高清 | 曰本女人与公拘交酡免费视频 | 国产精品怡红院永久免费 | 最近的中文字幕在线看视频 | 国产在线一区二区三区四区五区 | 中文字幕无码av激情不卡 | 国产另类ts人妖一区二区 | 少妇性l交大片欧洲热妇乱xxx | 国产精品人人爽人人做我的可爱 | 国内丰满熟女出轨videos | 国产欧美亚洲精品a | 国产精品第一国产精品 | 一本久久a久久精品亚洲 | 蜜桃臀无码内射一区二区三区 | 四虎国产精品免费久久 | 日韩精品无码免费一区二区三区 | 精品夜夜澡人妻无码av蜜桃 | 亚洲毛片av日韩av无码 | 国产在线精品一区二区三区直播 | 一本久久伊人热热精品中文字幕 | 久久午夜无码鲁丝片午夜精品 | 强伦人妻一区二区三区视频18 | 熟女少妇人妻中文字幕 | 亚洲精品综合一区二区三区在线 | 东京热一精品无码av | 亚无码乱人伦一区二区 | 97人妻精品一区二区三区 | 亚洲高清偷拍一区二区三区 | 久久综合激激的五月天 | 精品夜夜澡人妻无码av蜜桃 | 久久综合给合久久狠狠狠97色 | 亚洲第一网站男人都懂 | 国产熟妇另类久久久久 | 久久伊人色av天堂九九小黄鸭 | 性生交大片免费看l | 夜夜夜高潮夜夜爽夜夜爰爰 | 成人无码精品1区2区3区免费看 | 性欧美videos高清精品 | 无码任你躁久久久久久久 | 熟女少妇在线视频播放 | 欧美日韩一区二区免费视频 | 免费观看的无遮挡av | 一个人看的www免费视频在线观看 | 亚洲精品成人福利网站 | 又湿又紧又大又爽a视频国产 | 丰满人妻被黑人猛烈进入 | 久久亚洲精品中文字幕无男同 | 天堂亚洲免费视频 | 人妻互换免费中文字幕 | 婷婷五月综合激情中文字幕 | 熟女俱乐部五十路六十路av | 色欲av亚洲一区无码少妇 | 国产莉萝无码av在线播放 | 秋霞特色aa大片 | 中文字幕精品av一区二区五区 | 亚洲s色大片在线观看 | 久青草影院在线观看国产 | 久久久久久久久蜜桃 | 午夜福利试看120秒体验区 | 中文字幕无码日韩欧毛 | 国产免费久久精品国产传媒 | 人人澡人摸人人添 | 国内揄拍国内精品人妻 | 亚洲精品一区二区三区四区五区 | 性欧美牲交在线视频 | 丁香花在线影院观看在线播放 | 国产国语老龄妇女a片 | 无码福利日韩神码福利片 | 国产欧美精品一区二区三区 | 日本大香伊一区二区三区 | 亚洲精品一区二区三区大桥未久 | 人人爽人人澡人人人妻 | 强伦人妻一区二区三区视频18 | 最近中文2019字幕第二页 | 国产特级毛片aaaaaa高潮流水 | 少妇无码av无码专区在线观看 | www一区二区www免费 | 亚洲爆乳精品无码一区二区三区 | 一本色道婷婷久久欧美 | 波多野结衣av在线观看 | 国产农村乱对白刺激视频 | 国产精品-区区久久久狼 | 中文无码成人免费视频在线观看 | 一区二区三区高清视频一 | 伊在人天堂亚洲香蕉精品区 | 狠狠色噜噜狠狠狠狠7777米奇 | 国产做国产爱免费视频 | 欧美自拍另类欧美综合图片区 | 欧洲精品码一区二区三区免费看 | 人人妻人人藻人人爽欧美一区 | 久久精品人人做人人综合 | 日本熟妇乱子伦xxxx | 日日噜噜噜噜夜夜爽亚洲精品 | 亚洲乱亚洲乱妇50p | 国产电影无码午夜在线播放 | 中文字幕av无码一区二区三区电影 | 中文无码精品a∨在线观看不卡 | 国内精品久久毛片一区二区 | 东京一本一道一二三区 | 中文字幕久久久久人妻 | 亚洲综合色区中文字幕 | 日本护士xxxxhd少妇 | 亚洲男人av香蕉爽爽爽爽 | 国产精品久久精品三级 | 免费乱码人妻系列无码专区 | 一二三四在线观看免费视频 | 久久99精品久久久久婷婷 | 一本大道久久东京热无码av | 少妇久久久久久人妻无码 | 乱人伦人妻中文字幕无码 | 在线观看国产一区二区三区 | 色综合久久88色综合天天 | 狠狠cao日日穞夜夜穞av | 人人妻人人澡人人爽欧美一区九九 | 国产成人人人97超碰超爽8 | 一区二区传媒有限公司 | 高潮毛片无遮挡高清免费 | 无码精品人妻一区二区三区av | 成人无码精品一区二区三区 | 日韩人妻系列无码专区 | 亚洲成a人片在线观看无码3d | 77777熟女视频在线观看 а天堂中文在线官网 | 久久亚洲日韩精品一区二区三区 | 精品国偷自产在线 | 欧美日韩在线亚洲综合国产人 | 亚洲七七久久桃花影院 | 亚洲国产精品毛片av不卡在线 | 无套内射视频囯产 | 少妇性l交大片 | 丰满少妇弄高潮了www | 奇米影视888欧美在线观看 | 精品一区二区三区无码免费视频 | 亚洲精品中文字幕 | 亚洲春色在线视频 | 国产在线aaa片一区二区99 | 成人精品视频一区二区 | 国产精品视频免费播放 | 装睡被陌生人摸出水好爽 | 国产av无码专区亚洲a∨毛片 | 少妇高潮一区二区三区99 | 在线天堂新版最新版在线8 | 久久久国产一区二区三区 | 日日干夜夜干 | 欧美xxxxx精品 | 亚洲 欧美 激情 小说 另类 | 亚洲日韩av片在线观看 | 欧美一区二区三区 | 99视频精品全部免费免费观看 | 久久精品女人的天堂av | 国产精品对白交换视频 | 网友自拍区视频精品 | 男人的天堂av网站 | 久久综合九色综合97网 | 免费网站看v片在线18禁无码 | 野外少妇愉情中文字幕 | 又粗又大又硬又长又爽 | 又紧又大又爽精品一区二区 | 亚洲中文字幕av在天堂 | 少妇一晚三次一区二区三区 | 免费国产黄网站在线观看 | 国产精品高潮呻吟av久久 | 曰本女人与公拘交酡免费视频 | 性开放的女人aaa片 | 清纯唯美经典一区二区 | 中文字幕无码免费久久9一区9 | 欧美丰满熟妇xxxx | 国产精品手机免费 | 国产超级va在线观看视频 | 欧美日韩一区二区三区自拍 | 中文字幕人成乱码熟女app | 麻豆人妻少妇精品无码专区 | 国产99久久精品一区二区 | 亚洲另类伦春色综合小说 | 激情亚洲一区国产精品 | 国产一精品一av一免费 | 精品厕所偷拍各类美女tp嘘嘘 | 性色欲情网站iwww九文堂 | 久久午夜夜伦鲁鲁片无码免费 | 久久久无码中文字幕久... | 中文无码伦av中文字幕 | 双乳奶水饱满少妇呻吟 | 亚洲中文字幕久久无码 | 精品久久久无码人妻字幂 | 久久精品99久久香蕉国产色戒 | 天天爽夜夜爽夜夜爽 | 理论片87福利理论电影 | 一本色道久久综合亚洲精品不卡 | 国产亚洲欧美日韩亚洲中文色 | 成人影院yy111111在线观看 | 日日躁夜夜躁狠狠躁 | 一本色道久久综合狠狠躁 | 精品偷自拍另类在线观看 | 精品久久久无码中文字幕 | 久久无码中文字幕免费影院蜜桃 | 成人性做爰aaa片免费看 | 女人色极品影院 | 国产区女主播在线观看 | 中文字幕乱码中文乱码51精品 | 亚洲日韩一区二区三区 | 亚洲欧美综合区丁香五月小说 | 国产综合色产在线精品 | 一本色道久久综合亚洲精品不卡 | 成人动漫在线观看 | 激情内射日本一区二区三区 | 国产农村妇女aaaaa视频 撕开奶罩揉吮奶头视频 | 乱人伦中文视频在线观看 | 成人性做爰aaa片免费看不忠 | 2019nv天堂香蕉在线观看 | 亚洲熟妇色xxxxx亚洲 | 精品国产av色一区二区深夜久久 | 日日麻批免费40分钟无码 | 国产精品久久久久无码av色戒 | 久久人人爽人人人人片 | 亚洲熟熟妇xxxx | 麻豆果冻传媒2021精品传媒一区下载 | 99久久精品无码一区二区毛片 | 国产av无码专区亚洲awww | 欧美国产亚洲日韩在线二区 | 精品偷拍一区二区三区在线看 | 一本久久a久久精品亚洲 | 国精产品一品二品国精品69xx | 亚洲区欧美区综合区自拍区 | 国产亚av手机在线观看 | 麻豆蜜桃av蜜臀av色欲av | 四虎国产精品一区二区 | 国产精品99久久精品爆乳 | 国产成人一区二区三区别 | 国产精品嫩草久久久久 | 中文字幕无码av波多野吉衣 | 人人爽人人澡人人高潮 | 国产精品永久免费视频 | 国产成人av免费观看 | 亚洲精品成a人在线观看 | 乱码av麻豆丝袜熟女系列 | 久久综合给合久久狠狠狠97色 | 国产成人精品一区二区在线小狼 | 亚洲成a人片在线观看日本 | 丰满少妇女裸体bbw | 中文字幕+乱码+中文字幕一区 | 人人妻人人澡人人爽欧美精品 | 熟妇人妻激情偷爽文 | 欧美性生交xxxxx久久久 | 亚洲成a人片在线观看无码3d | 55夜色66夜色国产精品视频 | 精品欧美一区二区三区久久久 | 久久综合久久自在自线精品自 | 97se亚洲精品一区 | 日产精品99久久久久久 | 国内精品九九久久久精品 | 成人亚洲精品久久久久 | 亚洲 欧美 激情 小说 另类 | 亚洲aⅴ无码成人网站国产app | 国产成人无码一二三区视频 | 亚洲娇小与黑人巨大交 | 香蕉久久久久久av成人 | 色综合久久中文娱乐网 | 婷婷丁香六月激情综合啪 | 人妻互换免费中文字幕 | 国精品人妻无码一区二区三区蜜柚 | 成年美女黄网站色大免费全看 | 亚洲人成人无码网www国产 | 国产热a欧美热a在线视频 | 四虎影视成人永久免费观看视频 | 色一情一乱一伦一区二区三欧美 | 无码人中文字幕 | 国产熟妇高潮叫床视频播放 | 丰满人妻翻云覆雨呻吟视频 | 一本无码人妻在中文字幕免费 | 成人精品天堂一区二区三区 | 蜜桃视频韩日免费播放 | 精品久久久久香蕉网 | 欧美性生交xxxxx久久久 | 亚无码乱人伦一区二区 | а√天堂www在线天堂小说 | 澳门永久av免费网站 | 精品乱子伦一区二区三区 | 婷婷色婷婷开心五月四房播播 | 99久久精品国产一区二区蜜芽 | 99精品无人区乱码1区2区3区 | 精品厕所偷拍各类美女tp嘘嘘 | 无套内谢的新婚少妇国语播放 | 色诱久久久久综合网ywww | 香港三级日本三级妇三级 | 黑森林福利视频导航 | 高清无码午夜福利视频 | 妺妺窝人体色www在线小说 | 噜噜噜亚洲色成人网站 | 日本乱偷人妻中文字幕 | 粗大的内捧猛烈进出视频 | 成人毛片一区二区 | 国产成人无码区免费内射一片色欲 | 少妇人妻av毛片在线看 | 欧美三级不卡在线观看 | 亚洲精品一区二区三区在线 | 亚洲阿v天堂在线 | 久久99精品久久久久婷婷 | 精品国产aⅴ无码一区二区 | 欧美人与动性行为视频 | 无码一区二区三区在线 | 国产精品久久精品三级 | 一本久道久久综合婷婷五月 | 亚洲精品鲁一鲁一区二区三区 | 中文字幕人妻无码一夲道 | 亚洲国产欧美日韩精品一区二区三区 | 女人色极品影院 | 玩弄人妻少妇500系列视频 | 俄罗斯老熟妇色xxxx | 麻豆蜜桃av蜜臀av色欲av | 中文字幕乱妇无码av在线 | 国产精品二区一区二区aⅴ污介绍 | 亚洲gv猛男gv无码男同 | 好爽又高潮了毛片免费下载 |