當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

Spark学习之路（十五）SparkCore的源码解读（一）启动脚本

發布時間：2025/7/14 编程问答 24 豆豆

生活随笔收集整理的這篇文章主要介紹了 Spark学习之路（十五）SparkCore的源码解读（一）启动脚本小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

討論QQ：1586558083

一、啟動腳本分析
- 1.1　start-all.sh?
- 1.2　start-master.sh
- 1.3　spark-config.sh(1.2的第5步)
- 1.4　load-spark-env.sh(1.2的第6步)
- 1.5　spark-env.sh
- 1.6　spark-daemon.sh
- 1.7　spark-class
- 1.8　start-slaves.sh
- 1.9　轉向start-slave.sh
二、其他腳本
- 2.1　start-history-server.sh
- 2.2　start-shuffle-service.sh
- 2.3　start-thriftserver.sh

正文

回到頂部

一、啟動腳本分析

獨立部署模式下，主要由master和slaves組成，master可以利用zk實現高可用性，其driver，work，app等信息可以持久化到zk上；slaves由一臺至多臺主機構成。Driver通過向Master申請資源獲取運行環境。

啟動master和slaves主要是執行/usr/dahua/spark/sbin目錄下的start-master.sh和start-slaves.sh，或者執行

start-all.sh，其中star-all.sh本質上就是調用start-master.sh和start-slaves.sh

1.1　start-all.sh?

其中start-master.sh和start-slave.sh分別調用的是

org.apache.spark.deploy.master.Master和org.apache.spark.deploy.worker.Worker

1.2　start-master.sh

start-master.sh調用了spark-daemon.sh，注意這里指定了啟動的類

#1.判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; thenexport SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" fi# NOTE: This exact class name is matched downstream by SparkSubmit. # Any changes need to be reflected there. #2.設置CLASS="org.apache.spark.deploy.master.Master" CLASS="org.apache.spark.deploy.master.Master" #3.如果參數結尾包含--help或者-h則打印幫助信息，并退出 if [[ "$@" = *--help ]] || [[ "$@" = *-h ]]; then echo "Usage: ./sbin/start-master.sh [options]" pattern="Usage:" pattern+="\|Using Spark's default log4j profile:" pattern+="\|Registered signal handlers for" "${SPARK_HOME}"/bin/spark-class $CLASS --help 2>&1 | grep -v "$pattern" 1>&2 exit 1 fi #4.設置ORIGINAL_ARGS為所有參數 ORIGINAL_ARGS="$@" #5.執行${SPARK_HOME}/sbin/spark-config.sh . "${SPARK_HOME}/sbin/spark-config.sh" #6.執行${SPARK_HOME}/bin/load-spark-env.sh . "${SPARK_HOME}/bin/load-spark-env.sh" #7.SPARK_MASTER_PORT為空則賦值7077 if [ "$SPARK_MASTER_PORT" = "" ]; then SPARK_MASTER_PORT=7077 fi #8.SPARK_MASTER_HOST為空則賦值本主機名(hostname) if [ "$SPARK_MASTER_HOST" = "" ]; then case `uname` in (SunOS) SPARK_MASTER_HOST="`/usr/sbin/check-hostname | awk '{print $NF}'`" ;; (*) SPARK_MASTER_HOST="`hostname -f`" ;; esac fi #9.SPARK_MASTER_WEBUI_PORT為空則賦值8080 if [ "$SPARK_MASTER_WEBUI_PORT" = "" ]; then SPARK_MASTER_WEBUI_PORT=8080 fi #10.執行腳本 "${SPARK_HOME}/sbin"/spark-daemon.sh start $CLASS 1 \ --host $SPARK_MASTER_HOST --port $SPARK_MASTER_PORT --webui-port $SPARK_MASTER_WEBUI_PORT \ $ORIGINAL_ARGS

其中10肯定是重點，分析之前我們看看5，6都干了些啥，最后直譯出最后一個腳本

1.3　spark-config.sh(1.2的第5步)

#判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; thenexport SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" fi #SPARK_CONF_DIR存在就用此目錄，不存在用${SPARK_HOME}/conf export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}/conf"}" # Add the PySpark classes to the PYTHONPATH: if [ -z "${PYSPARK_PYTHONPATH_SET}" ]; then export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}" export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.6-src.zip:${PYTHONPATH}" export PYSPARK_PYTHONPATH_SET=1 fi

1.4　load-spark-env.sh(1.2的第6步)

#1.判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; thensource "$(dirname "$0")"/find-spark-home fi #2.判斷SPARK_ENV_LOADED是否有值，沒有將其設置為1 if [ -z "$SPARK_ENV_LOADED" ]; then export SPARK_ENV_LOADED=1 #3.設置user_conf_dir為SPARK_CONF_DIR或SPARK_HOME/conf export SPARK_CONF_DIR="${SPARK_CONF_DIR:-"${SPARK_HOME}"/conf}" #4.執行"${user_conf_dir}/spark-env.sh" [注：set -/+a含義再做研究] if [ -f "${SPARK_CONF_DIR}/spark-env.sh" ]; then # Promote all variable declarations to environment (exported) variables set -a . "${SPARK_CONF_DIR}/spark-env.sh" set +a fi fi # Setting SPARK_SCALA_VERSION if not already set. #5.選擇scala版本，2.11和2.12都存在的情況下，優先選擇2.11 if [ -z "$SPARK_SCALA_VERSION" ]; then ASSEMBLY_DIR2="${SPARK_HOME}/assembly/target/scala-2.11" ASSEMBLY_DIR1="${SPARK_HOME}/assembly/target/scala-2.12" if [[ -d "$ASSEMBLY_DIR2" && -d "$ASSEMBLY_DIR1" ]]; then echo -e "Presence of build for multiple Scala versions detected." 1>&2 echo -e 'Either clean one of them or, export SPARK_SCALA_VERSION in spark-env.sh.' 1>&2 exit 1 fi if [ -d "$ASSEMBLY_DIR2" ]; then export SPARK_SCALA_VERSION="2.11" else export SPARK_SCALA_VERSION="2.12" fi fi

1.5　spark-env.sh

列舉很多種模式的選項配置

1.6　spark-daemon.sh

回過頭來看看1.2第10步中需要直譯出的最后一個腳本,如下：

sbin/spark-daemon.sh start org.apache.spark.deploy.master.Master 1 --host hostname --port 7077 --webui-port 8080

上面搞了半天只是設置了變量，最終才進入主角，繼續分析spark-daemon.sh腳本

#1.參數個數小于等于1，打印幫助 if [ $# -le 1 ]; thenecho $usageexit 1 fi #2.判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; then export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" fi #3.執行${SPARK_HOME}/sbin/spark-config.sh，見上述分析 [類似腳本是否有重復？原因是有的人是直接用spark-daemon.sh啟動的服務，反正重復設置下變量不需要什么代價] . "${SPARK_HOME}/sbin/spark-config.sh" # get arguments # Check if --config is passed as an argument. It is an optional parameter. # Exit if the argument is not a directory. #4.判斷第一個參數是否是--config,如果是取空格后一個字符串，然后判斷該目錄是否存在，不存在則打印錯誤信息并退出，存在設置SPARK_CONF_DIR為該目錄,shift到下一個參數 #[注：--config只能用在第一參數上] if [ "$1" == "--config" ] then shift conf_dir="$1" if [ ! -d "$conf_dir" ] then echo "ERROR : $conf_dir is not a directory" echo $usage exit 1 else export SPARK_CONF_DIR="$conf_dir" fi shift fi #5.分別設置option、command、instance為后面的三個參數(如：option=start,command=org.apache.spark.deploy.master.Master,instance=1) #[注：很多人用spark-daemon.sh啟動服務不成功的原因是名字不全] option=$1 shift command=$1 shift instance=$1 shift #6.日志回滾函數，主要用于更改日志名，如log-->log.1等，略過 spark_rotate_log () { log=$1; num=5; if [ -n "$2" ]; then num=$2 fi if [ -f "$log" ]; then # rotate logs while [ $num -gt 1 ]; do prev=`expr $num - 1` [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num" num=$prev done mv "$log" "$log.$num"; fi } #7.執行${SPARK_HOME}/bin/load-spark-env.sh，見上述分析 . "${SPARK_HOME}/bin/load-spark-env.sh" #8.判斷SPARK_IDENT_STRING是否有值，沒有將其設置為$USER(linux用戶) if [ "$SPARK_IDENT_STRING" = "" ]; then export SPARK_IDENT_STRING="$USER" fi #9.設置SPARK_PRINT_LAUNCH_COMMAND=1 export SPARK_PRINT_LAUNCH_COMMAND="1" # get log directory #10.判斷SPARK_LOG_DIR是否有值，沒有將其設置為${SPARK_HOME}/logs，并創建改目錄，測試創建文件，修改權限 if [ "$SPARK_LOG_DIR" = "" ]; then export SPARK_LOG_DIR="${SPARK_HOME}/logs" fi mkdir -p "$SPARK_LOG_DIR" touch "$SPARK_LOG_DIR"/.spark_test > /dev/null 2>&1 TEST_LOG_DIR=$? if [ "${TEST_LOG_DIR}" = "0" ]; then rm -f "$SPARK_LOG_DIR"/.spark_test else chown "$SPARK_IDENT_STRING" "$SPARK_LOG_DIR" fi #11.判斷SPARK_PID_DIR是否有值，沒有將其設置為/tmp if [ "$SPARK_PID_DIR" = "" ]; then SPARK_PID_DIR=/tmp fi # some variables #12.設置log和pid log="$SPARK_LOG_DIR/spark-$SPARK_IDENT_STRING-$command-$instance-$HOSTNAME.out" pid="$SPARK_PID_DIR/spark-$SPARK_IDENT_STRING-$command-$instance.pid" # Set default scheduling priority #13.判斷SPARK_NICENESS是否有值，沒有將其設置為0 [注：調度優先級，見后面] if [ "$SPARK_NICENESS" = "" ]; then export SPARK_NICENESS=0 fi #14.execute_command()函數，暫且略過，調用時再作分析 execute_command() { if [ -z ${SPARK_NO_DAEMONIZE+set} ]; then nohup -- "$@" >> $log 2>&1 < /dev/null & newpid="$!" echo "$newpid" > "$pid" # Poll for up to 5 seconds for the java process to start for i in {1..10} do if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then break fi sleep 0.5 done sleep 2 # Check if the process has died; in that case we'll tail the log so the user can see if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then echo "failed to launch: $@" tail -10 "$log" | sed 's/^/ /' echo "full log in $log" fi else "$@" fi } #15.進入case語句，判斷option值，進入該分支，我們以start為例 # 執行run_command class "$@"，其中$@此時為空，經驗證，啟動帶上此參數后，關閉也需，不然關閉不了，后面再分析此參數作用 # 我們正式進入run_command()函數，分析 # I.設置mode=class,創建SPARK_PID_DIR，上面的pid文件是否存在， # II.SPARK_MASTER不為空，同步刪除某些文件 # III.回滾log日志 # IV.進入case，command=org.apache.spark.deploy.master.Master，最終執行 # nohup nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class $command "$@" >> "$log" 2>&1 < /dev/null & # newpid="$!" # echo "$newpid" > "$pid" # 重點轉向bin/spark-class org.apache.spark.deploy.master.Master run_command() { mode="$1" shift mkdir -p "$SPARK_PID_DIR" if [ -f "$pid" ]; then TARGET_ID="$(cat "$pid")" if [[ $(ps -p "$TARGET_ID" -o comm=) =~ "java" ]]; then echo "$command running as process $TARGET_ID. Stop it first." exit 1 fi fi if [ "$SPARK_MASTER" != "" ]; then echo rsync from "$SPARK_MASTER" rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' "$SPARK_MASTER/" "${SPARK_HOME}" fi spark_rotate_log "$log" echo "starting $command, logging to $log" case "$mode" in (class) execute_command nice -n "$SPARK_NICENESS" "${SPARK_HOME}"/bin/spark-class "$command" "$@" ;; (submit) execute_command nice -n "$SPARK_NICENESS" bash "${SPARK_HOME}"/bin/spark-submit --class "$command" "$@" ;; (*) echo "unknown mode: $mode" exit 1 ;; esac } case $option in (submit) run_command submit "$@" ;; (start) run_command class "$@" ;; (stop) if [ -f $pid ]; then TARGET_ID="$(cat "$pid")" if [[ $(ps -p "$TARGET_ID" -o comm=) =~ "java" ]]; then echo "stopping $command" kill "$TARGET_ID" && rm -f "$pid" else echo "no $command to stop" fi else echo "no $command to stop" fi ;; (status) if [ -f $pid ]; then TARGET_ID="$(cat "$pid")" if [[ $(ps -p "$TARGET_ID" -o comm=) =~ "java" ]]; then echo $command is running. exit 0 else echo $pid file is present but $command not running exit 1 fi else echo $command not running. exit 2 fi ;; (*) echo $usage exit 1 ;; esac

1.7　spark-class

#1.判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; thensource "$(dirname "$0")"/find-spark-home fi#2.執行${SPARK_HOME}/bin/load-spark-env.sh，見上述分析 . "${SPARK_HOME}"/bin/load-spark-env.sh # Find the java binary #3.判斷JAVA_HOME是否為NULL，不是則設置RUNNER="${JAVA_HOME}/bin/java"，否則找系統自帶，在沒有則報未設置，并退出 if [ -n "${JAVA_HOME}" ]; then RUNNER="${JAVA_HOME}/bin/java" else if [ "$(command -v java)" ]; then RUNNER="java" else echo "JAVA_HOME is not set" >&2 exit 1 fi fi # Find Spark jars. #4.查找SPARK_JARS_DIR，若${SPARK_HOME}/RELEASE文件存在，則SPARK_JARS_DIR="${SPARK_HOME}/jars"，否則 #SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars" if [ -d "${SPARK_HOME}/jars" ]; then SPARK_JARS_DIR="${SPARK_HOME}/jars" else SPARK_JARS_DIR="${SPARK_HOME}/assembly/target/scala-$SPARK_SCALA_VERSION/jars" fi #5.若SPARK_JARS_DIR不存在且$SPARK_TESTING$SPARK_SQL_TESTING有值[注：一般我們不設置這兩變量]，報錯退出，否則LAUNCH_CLASSPATH="$SPARK_JARS_DIR/*" if [ ! -d "$SPARK_JARS_DIR" ] && [ -z "$SPARK_TESTING$SPARK_SQL_TESTING" ]; then echo "Failed to find Spark jars directory ($SPARK_JARS_DIR)." 1>&2 echo "You need to build Spark with the target \"package\" before running this program." 1>&2 exit 1 else LAUNCH_CLASSPATH="$SPARK_JARS_DIR/*" fi # Add the launcher build dir to the classpath if requested. #6.SPARK_PREPEND_CLASSES不是NULL，則LAUNCH_CLASSPATH="${SPARK_HOME}/launcher/target/scala-$SPARK_SCALA_VERSION/classes:$LAUNCH_CLASSPATH"， #添加編譯相關至LAUNCH_CLASSPATH if [ -n "$SPARK_PREPEND_CLASSES" ]; then LAUNCH_CLASSPATH="${SPARK_HOME}/launcher/target/scala-$SPARK_SCALA_VERSION/classes:$LAUNCH_CLASSPATH" fi # For tests #7.SPARK_TESTING不是NULL，則unset YARN_CONF_DIR和unset HADOOP_CONF_DIR，暫且當做是為了某種測試 if [[ -n "$SPARK_TESTING" ]]; then unset YARN_CONF_DIR unset HADOOP_CONF_DIR fi #8.build_command函數，略過 build_command() { "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@" printf "%d\0" $? } # Turn off posix mode since it does not allow process substitution set +o posix CMD=() while IFS= read -d '' -r ARG; do CMD+=("$ARG") #9.最終調用"$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"， #直譯：java -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@" #轉向java類org.apache.spark.launcher.Main，這就是java入口類 done < <(build_command "$@") COUNT=${#CMD[@]} LAST=$((COUNT - 1)) LAUNCHER_EXIT_CODE=${CMD[$LAST]} # Certain JVM failures result in errors being printed to stdout (instead of stderr), which causes # the code that parses the output of the launcher to get confused. In those cases, check if the # exit code is an integer, and if it's not, handle it as a special error case. if ! [[ $LAUNCHER_EXIT_CODE =~ ^[0-9]+$ ]]; then echo "${CMD[@]}" | head -n-1 1>&2 exit 1 fi if [ $LAUNCHER_EXIT_CODE != 0 ]; then exit $LAUNCHER_EXIT_CODE fi CMD=("${CMD[@]:0:$LAST}") exec "${CMD[@]}"

1.8　start-slaves.sh

#1.判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; thenexport SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" fi#2.執行${SPARK_HOME}/sbin/spark-config.sh，見上述分析 . "${SPARK_HOME}/sbin/spark-config.sh" #3.執行${SPARK_HOME}/bin/load-spark-env.sh，見上述分析 . "${SPARK_HOME}/bin/load-spark-env.sh" # Find the port number for the master #4.SPARK_MASTER_PORT為空則設置為7077 if [ "$SPARK_MASTER_PORT" = "" ]; then SPARK_MASTER_PORT=7077 fi #5.SPARK_MASTER_HOST為空則設置為`hostname` if [ "$SPARK_MASTER_HOST" = "" ]; then case `uname` in (SunOS) SPARK_MASTER_HOST="`/usr/sbin/check-hostname | awk '{print $NF}'`" ;; (*) SPARK_MASTER_HOST="`hostname -f`" ;; esac fi # Launch the slaves #6.啟動slaves， # "${SPARK_HOME}/sbin/slaves.sh" cd "${SPARK_HOME}" \; "${SPARK_HOME}/sbin/start-slave.sh" "spark://$SPARK_MASTER_HOST:$SPARK_MASTER_PORT" # 遍歷conf/slaves中主機，其中有設置SPARK_SSH_OPTS，ssh每一臺機器執行"${SPARK_HOME}/sbin/start-slave.sh" "spark://$SPARK_MASTER_HOST:$SPARK_MASTER_PORT" "${SPARK_HOME}/sbin/slaves.sh" cd "${SPARK_HOME}" \; "${SPARK_HOME}/sbin/start-slave.sh" "spark://$SPARK_MASTER_HOST:$SPARK_MASTER_PORT"

1.9　轉向start-slave.sh

#1.判斷SPARK_HOME是否有值，沒有將其設置為當前文件所在目錄的上級目錄 if [ -z "${SPARK_HOME}" ]; thenexport SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)" fi#2.設置CLASS="org.apache.spark.deploy.worker.Worker" CLASS="org.apache.spark.deploy.worker.Worker" #3.如果參數結尾包含--help或者-h則打印幫助信息，并退出 if [[ $# -lt 1 ]] || [[ "$@" = *--help ]] || [[ "$@" = *-h ]]; then echo "Usage: ./sbin/start-slave.sh [options] <master>" pattern="Usage:" pattern+="\|Using Spark's default log4j profile:" pattern+="\|Registered signal handlers for" "${SPARK_HOME}"/bin/spark-class $CLASS --help 2>&1 | grep -v "$pattern" 1>&2 exit 1 fi #4.執行${SPARK_HOME}/sbin/spark-config.sh，見上述分析 . "${SPARK_HOME}/sbin/spark-config.sh" #5.執行${SPARK_HOME}/bin/load-spark-env.sh，見上述分析 . "${SPARK_HOME}/bin/load-spark-env.sh" #6.MASTER=$1,這里MASTER=spark://hostname:7077，然后shift，也就是說單獨啟動單個slave使用start-slave.sh spark://hostname:7077 MASTER=$1 shift #7.SPARK_WORKER_WEBUI_PORT為空則設置為8081 if [ "$SPARK_WORKER_WEBUI_PORT" = "" ]; then SPARK_WORKER_WEBUI_PORT=8081 fi #8.函數start_instance，略過 function start_instance { #設置WORKER_NUM=$1 WORKER_NUM=$1 shift if [ "$SPARK_WORKER_PORT" = "" ]; then PORT_FLAG= PORT_NUM= else PORT_FLAG="--port" PORT_NUM=$(( $SPARK_WORKER_PORT + $WORKER_NUM - 1 )) fi WEBUI_PORT=$(( $SPARK_WORKER_WEBUI_PORT + $WORKER_NUM - 1 )) #直譯：spark-daemon.sh start org.apache.spark.deploy.worker.Worker 1 --webui-port 7077 spark://hostname:7077 #代碼再次轉向spark-daemon.sh，見上訴分析 "${SPARK_HOME}/sbin"/spark-daemon.sh start $CLASS $WORKER_NUM \ --webui-port "$WEBUI_PORT" $PORT_FLAG $PORT_NUM $MASTER "$@" } #9.判斷SPARK_WORKER_INSTANCES(可以認為是單節點Worker進程數)是否為空 # 為空，則start_instance 1 "$@" # 不為空，則循環 # for ((i=0; i<$SPARK_WORKER_INSTANCES; i++)); do # start_instance $(( 1 + $i )) "$@" # done if [ "$SPARK_WORKER_INSTANCES" = "" ]; then start_instance 1 "$@" else for ((i=0; i<$SPARK_WORKER_INSTANCES; i++)); do #10.轉向start_instance函數 start_instance $(( 1 + $i )) "$@" done fi

回到頂部

二、其他腳本

2.1　start-history-server.sh

2.2　start-shuffle-service.sh

2.3　start-thriftserver.sh

開啟thriftserver，略

轉載于:https://www.cnblogs.com/liuys635/p/11002749.html

總結

以上是生活随笔為你收集整理的Spark学习之路（十五）SparkCore的源码解读（一）启动脚本的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。

上一篇： SpringCloud(一)：深入理解E
下一篇：如何向妻子解释OOD

编程问答

Spark学习之路 （十五）SparkCore的源码解读（一）启动脚本

一、啟動腳本分析

1.1 start-all.sh?

1.2 start-master.sh

1.3 spark-config.sh(1.2的第5步)

1.4 load-spark-env.sh(1.2的第6步)

1.5 spark-env.sh

1.6 spark-daemon.sh

1.7 spark-class

1.8 start-slaves.sh

1.9 轉向start-slave.sh