步驟:

①$FLINK_HOME/bin/yarn-session.sh

②$FLINK_HOME/bin/start-scala-shell.sh yarn

注意哈，和提交flink任務不同，不存在 per-job模式的flink on yarn shell這種啟動方式。

per-job的提交方式示例[1]:

bin/flink run -m yarn-cluster -yn 3 -ys 3 -ynm bjsxt02 -c com.test.flink.wc.StreamWordCount ./appjars/test-1.0-SNAPSHOT.jar

[1]中同時提到:

這種模式，每次提交 Flink 任務都會創(chuàng)建一個新的 Flink 集群，每個 Flink 任務之間相互獨立、互不影響，管理方便。任務執(zhí)行完成之后創(chuàng)建的 Flink集群也會消失，不會額外占用資源，按需使用，這使資源利用率達到最大，在工作中推薦使用這種模式。

?

說跑題了，下面繼續(xù)正題:

?

DataStream例子:

val textStreaming = senv.fromElements("To be, or not to be,--that is the question:--") val countsStreaming = textStreaming.flatMap { _.toLowerCase.split("\\W+") }.map { (_, 1) }.keyBy(0).sum(1) countsStreaming.print() senv.execute("Streaming Wordcount")

運行的log很長:

2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: jobmanager.rpc.address, Desktop 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: jobmanager.rpc.port, 6123 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: jobmanager.heap.size, 2048m 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: taskmanager.memory.process.size, 1728m 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: taskmanager.numberOfTaskSlots, 2 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: parallelism.default, 2 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: jobmanager.execution.failover-strategy, region 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: rest.port, 8082 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: rest.address, Desktop 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: web.submit.enable, true 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: jobmanager.archive.fs.dir, hdfs://Desktop:9000/completed-jobs/ 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: historyserver.web.address, Desktop 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: historyserver.web.port, 18082 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: historyserver.archive.fs.dir, hdfs://Desktop:9000/completed-jobs/ 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: historyserver.archive.fs.refresh-interval, 10000 2020-06-10 16:54:14,465 INFO configuration.GlobalConfiguration: Loading configuration property: env.java.home, /home/appleyuchi/Java/jdk1.8.0_131 2020-06-10 16:54:14,677 INFO client.RMProxy: Connecting to ResourceManager at Desktop/192.168.0.103:8032 2020-06-10 16:54:14,742 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2020-06-10 16:54:14,911 INFO client.AHSProxy: Connecting to Application History server at Desktop/192.168.0.103:10201 2020-06-10 16:54:14,920 INFO yarn.YarnClusterDescriptor: No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2020-06-10 16:54:14,996 INFO yarn.YarnClusterDescriptor: Found Web Interface laptop:8082 of application 'application_1591750252874_0011'. res1: org.apache.flink.api.common.JobExecutionResult = Program execution finished Job with JobID b965ad1132a622dc6b57de17ce7a1193 has finished. Job Runtime: 7097 ms

因為是flink on yarn模式，所以結(jié)果不會出現(xiàn)在$FLINK_HOME/log下面。

這種模式的實驗結(jié)果非常難找，在yarn界面上找不到,心累啊，唉，到底在哪里？

找了很久，最終結(jié)果在slave節(jié)點(我的是Laptop)中的$HADOOP_HOME/logs/userlogs下面：

(Python3.6) appleyuchi@Laptop:container_1591832727725_0001_01_000002$ pwd
/home/appleyuchi/bigdata/hadoop-3.1.2/logs/userlogs/application_1591832727725_0001/container_1591832727725_0001_01_000002
(Python3.6) appleyuchi@Laptop:container_1591832727725_0001_01_000002$ cat taskmanager.out
1> (not,1)
2> (to,1)
2> (be,1)
2> (or,1)
2> (to,2)
2> (be,2)
2> (that,1)
2> (is,1)
2> (the,1)
2> (question,1)

?

Reference:

[1]FLINK ON YARN 部署SESSION-CLUSTER和PER-JOB-CLUSTER

總結(jié)

以上是生活随笔為你收集整理的flink on yarn shell的session cluster模式实验记录的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。