Hadoop 的 PolyBase 配置和安全PolyBase configuration and security for Hadoop

04/23/2019

本文內容

適用于：Applies to: SQL ServerSQL Server(所有支持的版本)SQL ServerSQL Server (all supported versions) - 僅限 Windows Azure SQL 托管實例Azure SQL Managed InstanceAzure SQL 托管實例Azure SQL Managed Instance適用于：Applies to: SQL ServerSQL Server(所有支持的版本)SQL ServerSQL Server (all supported versions) - Windows only Azure SQL 托管實例Azure SQL Managed InstanceAzure SQL 托管實例Azure SQL Managed Instance

本文為影響 PolyBase 與 Hadoop 的連接的各種配置設置提供參考。This article provides a reference for various configuration settings that affect PolyBase connectivity to Hadoop. 有關如何將 PolyBase 與 Hadoop 配合使用的演練，請參閱配置 PolyBase 以訪問 Hadoop 中的外部數據。For a walkthrough on how to use PolyBase with Hadoop, see Configure PolyBase to access external data in Hadoop.

Hadoop.RPC.Protection 設置Hadoop.RPC.Protection setting

在 Hadoop 群集中保護通信的常用方法是將 hadoop.rpc.protection 配置更改為“隱私”或“完整性”。A common way to secure communication in a hadoop cluster is by changing the hadoop.rpc.protection configuration to 'Privacy' or 'Integrity'. 默認情況下，PolyBase 假定配置設置為“身份驗證”。By default, PolyBase assumes the configuration is set to 'Authenticate'. 要替代此默認設置，請將以下屬性添加到 core-site.xml 文件。To override this default, add the following property to the core-site.xml file. 通過更改此配置，可以實現 hadoop 節點之間的安全數據傳輸，以及 TLS 與 SQL Server 的連接。Changing this configuration will enable secure data transfer among the hadoop nodes and TLS connection to SQL Server.

hadoop.rpc.protection

若要對 hadoop.rpc.protection 使用“隱私”或“完整性”，SQL Server 必須至少為 SQL Server 2016 SP1 CU7、SQL Server 2016 SP2 或 SQL Server 2017 CU3。To use 'Privacy' or 'Integrity' for hadoop.rpc.protection, SQL Server must be at least SQL Server 2016 SP1 CU7, SQL Server 2016 SP2, or SQL Server 2017 CU3.

CDH 5.X 群集的示例 XML 文件Example XML files for CDH 5.X cluster

包含 yarn.application.classpath 和 mapreduce.application.classpath 配置的 Yarn-site.xml。Yarn-site.xml with yarn.application.classpath and mapreduce.application.classpath configuration.

yarn.resourcemanager.connect.max-wait.ms

40000

yarn.resourcemanager.connect.retry-interval.ms

30000

CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries

yarn.application.classpath

$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH*

如果你選擇將兩個配置設置拆分為 mapred-site.xml 和 yarn-site.xml，則這兩個文件將如下所示：If you choose to break your two configuration settings into the mapred-site.xml and the yarn-site.xml then the files would be the following:

yarn-site.xmlyarn-site.xml

yarn.resourcemanager.connect.max-wait.ms

40000

yarn.resourcemanager.connect.retry-interval.ms

30000

CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries

yarn.application.classpath

$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*

mapred-site.xmlmapred-site.xml

注意，我們添加了屬性 mapreduce.application.classpath。Note that we added the property mapreduce.application.classpath. 在 CDH 5.x 中，你會發現遵守 Ambari 中的相同命名約定的配置值。In CDH 5.x you will find the configuration values under the same naming convention in Ambari.

mapred.min.split.size

1073741824

mapreduce.app-submission.cross-platform

true

mapreduce.application.classpath

$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH

Kerberos 配置Kerberos configuration

請注意，當 PolyBase 向 Kerberos 保護的群集證明身份時，默認情況下需要將 hadoop.rpc.protection 設置設為“身份驗證”。Note, when PolyBase authenticates to a Kerberos secured cluster, it expects the hadoop.rpc.protection setting is 'Authenticate' by default. 這會使 Hadoop 節點間的數據通信保持非加密狀態。This leaves the data communication between Hadoop nodes unencrypted. 要為 hadoop.rpc.protection 使用“隱私”或“完整性”設置，請在 PolyBase 服務器上更新 core-site.xml 文件。To use 'Privacy' or 'Integrity' settings for hadoop.rpc.protection, update the core-site.xml file on the PolyBase server.

使用 MIT KDC 連接到 Kerberos 保護的 Hadoop 群集：To connect to a Kerberos-secured Hadoop cluster using MIT KDC:

在 SQL Server 的安裝路徑中查找 Hadoop 配置目錄。Find the Hadoop configuration directory in the installation path of SQL Server. 通常情況下，該路徑為：Typically, the path is:

C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn\PolyBase\Hadoop\conf

查找表中列出的配置密鑰 Hadoop 端配置值。Find the Hadoop side configuration value of the configuration keys listed in the table. (對于 Hadoop 計算機，在 Hadoop 配置目錄中查找文件。)(On the Hadoop machine, find the files in the Hadoop configuration directory.)

將配置值復制到 SQL Server 計算機上對應文件的值屬性中。Copy the configuration values into the value property in the corresponding files on the SQL Server machine.

#

配置文件Configuration file

配置密鑰Configuration key

ActionAction

11

core-site.xmlcore-site.xml

polybase.kerberos.kdchostpolybase.kerberos.kdchost

指定 KDC 主機名。Specify the KDC hostname. 例如：kerberos.your-realm.com。For example: kerberos.your-realm.com.

22

core-site.xmlcore-site.xml

polybase.kerberos.realmpolybase.kerberos.realm

指定 Kerberos 領域。Specify the Kerberos realm. 例如：YOUR-REALM.COMFor example: YOUR-REALM.COM

配置說明：領域名必須采用大寫形式。Configuration note: Realm name must be written in upper case.

33

core-site.xmlcore-site.xml

hadoop.security.authenticationhadoop.security.authentication

查找 Hadoop 端配置并復制到 SQL Server 計算機。Find the Hadoop side configuration and copy to SQL Server machine. 例如：KERBEROSFor example: KERBEROS

安全說明： KERBEROS 必須采用大寫形式。Security note: KERBEROS must be written in upper case. 如果采用小寫，則可能不會打開。If lower case, it might not be on.

44

hdfs-site.xmlhdfs-site.xml

dfs.namenode.kerberos.principaldfs.namenode.kerberos.principal

查找 Hadoop 端配置并復制到 SQL Server 計算機。Find the Hadoop side configuration and copy to SQL Server machine. 例如： hdfs/_HOST@YOUR-REALM.COMFor example: hdfs/_HOST@YOUR-REALM.COM

55

mapred-site.xmlmapred-site.xml

mapreduce.jobhistory.principalmapreduce.jobhistory.principal

查找 Hadoop 端配置并復制到 SQL Server 計算機。Find the Hadoop side configuration and copy to SQL Server machine. 例如： mapred/_HOST@YOUR-REALM.COMFor example: mapred/_HOST@YOUR-REALM.COM

66

mapred-site.xmlmapred-site.xml

mapreduce.jobhistory.addressmapreduce.jobhistory.address

查找 Hadoop 端配置并復制到 SQL Server 計算機。Find the Hadoop side configuration and copy to SQL Server machine. 例如：10.193.26.174:10020For example: 10.193.26.174:10020

77

yarn-site.xml yarn。yarn-site.xml yarn.

yarn.resourcemanager.principalyarn.resourcemanager.principal

查找 Hadoop 端配置并復制到 SQL Server 計算機。Find the Hadoop side configuration and copy to SQL Server machine. 例如： yarn/_HOST@YOUR-REALM.COMFor example: yarn/_HOST@YOUR-REALM.COM

創建數據庫范圍內的憑據對象，以指定每個 Hadoop 用戶的身份驗證信息。Create a database-scoped credential object to specify the authentication information for each Hadoop user.

后續步驟Next steps

有關詳細信息，請參閱以下文章：For more information, see the following articles:

總結

以上是生活随笔為你收集整理的polybase配置 sql_Hadoop 的 PolyBase 配置和安全的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。