在按照王家林的文檔安裝完scala，spark集群和idea-IC開發工具后，用spark自帶的示例SparkPi測試scala和spark集群

1、按照王家林文檔中的方法把spark自帶的SparkPi示例的代碼復制到idea-IC開發工具中創建的項目中后，把SparkPi代碼的修改為：

/*
?* Licensed to the Apache Software Foundation (ASF) under one or more
?* contributor license agreements. ?See the NOTICE file distributed with
?* this work for additional information regarding copyright ownership.
?* The ASF licenses this file to You under the Apache License, Version 2.0
?* (the "License"); you may not use this file except in compliance with
?* the License. ?You may obtain a copy of the License at
?*
?* ? ?http://www.apache.org/licenses/LICENSE-2.0
?*
?* Unless required by applicable law or agreed to in writing, software
?* distributed under the License is distributed on an "AS IS" BASIS,
?* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
?* See the License for the specific language governing permissions and
?* limitations under the License.
?*/
import scala.math.random
import org.apache.spark._

/** Computes an approximation to pi */
object SparkPi {
? def main(args: Array[String]) {
? ? val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://192.168.91.128:7077")//改成自己master主機的ip地址
? ? val spark = new SparkContext(conf)
? ? spark.addJar("/home/cy/workspace/untitled/out/artifacts/sparkpi_jar/untitled.jar")//把寫的代碼打包成jar包，存放打包后jar包的路徑
? ? val slices = if (args.length > 0) args(0).toInt else 2
? ? val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
? ? val count = spark.parallelize(1 until n, slices).map { i =>
? ? ? val x = random * 2 - 1
? ? ? val y = random * 2 - 1
? ? ? if (x*x + y*y < 1) 1 else 0
? ? }.reduce(_ + _)
? ? println("Pi is roughly " + 4.0 * count / n)
? ? spark.stop()
? }
}

2、點擊idea-IC開發工具左上角的File->Project Structure->Artifcats

3、添加jar包：

4、選擇Main Class，點擊OK：

5、修改jar包的名字、保存的路徑和類型，注意上述代碼中jar包的路徑要與這邊jar包的路徑一致：

6、完成上述的操作后，先通過start-all.sh啟動hadoop集群，再進入spark安裝目錄的sbin目錄下，通過./start-all.sh命令啟動spark集群

7、運行SparkPi代碼，出現以下結果就說明成功了：

8、要關掉集群的時候，先進入spark安裝目錄下的sbin目錄，通過./stop-all.sh命令先關掉spark集群，再通過stop-all.sh關掉hadoop集群

總結

以上是生活随笔為你收集整理的用spark自带的示例SparkPi测试scala和spark集群的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。