Argument

Meaning

-h?HOST,?--host?HOST

Hostname?to?listen?on

-i?HOST,?--ip?HOST

Hostname?to?listen?on?(deprecated,?use?-h?or?--host)

-p?PORT,?--port?PORT

Port?for?service?to?listen?on?(default:?7077?for?master,?random?for?worker)

--webui-port?PORT

Port?for?web?UI?(default:?8080?for?master,?8081?for?worker)

-c?CORES,?--cores?CORES

Total?CPU?cores?to?allow?Spark?applications?to?use?on?the?machine?(default:?all?available);?only?on?worker

-m?MEM,?--memory?MEM

Total?amount?of?memory?to?allow?Spark?applications?to?use?on?the?machine,?in?a?format?like?1000M?or?2G?(default:?your?machine's?total?RAM?minus?1?GB);?only?on?worker

-d?DIR,?--work-dir?DIR

Directory?to?use?for?scratch?space?and?job?output?logs?(default:?SPARK_HOME/work);?only?on?worker

--properties-file?FILE

Path?to?a?custom?Spark?properties?file?to?load?(default:?conf/spark-defaults.conf)

Environment?Variable

Meaning

SPARK_MASTER_IP

Bind?the?master?to?a?specific?IP?address,?for?example?a?public?one.

SPARK_MASTER_PORT

Start?the?master?on?a?different?port?(default:?7077).

SPARK_MASTER_WEBUI_PORT

Port?for?the?master?web?UI?(default:?8080).

SPARK_MASTER_OPTS

Configuration?properties?that?apply?only?to?the?master?in?the?form?"-Dx=y"?(default:?none).?See?below?for?a?list?of?possible?options.

SPARK_LOCAL_DIRS

Directory?to?use?for?"scratch"?space?in?Spark,?including?map?output?files?and?RDDs?that?get?stored?on?disk.?This?should?be?on?a?fast,?local?disk?in?your?system.?It?can?also?be?a?comma-separated?list?of?multiple?directories?on?different?disks.?

SPARK_WORKER_CORES

Total?number?of?cores?to?allow?Spark?applications?to?use?on?the?machine?(default:?all?available?cores).

SPARK_WORKER_MEMORY

Total?amount?of?memory?to?allow?Spark?applications?to?use?on?the?machine,?e.g.?1000m,?2g?(default:?total?memory?minus?1?GB);?note?that?each?application's?individual?memory?is?configured?using?its?spark.executor.memory?property.

SPARK_WORKER_PORT

Start?the?Spark?worker?on?a?specific?port?(default:?random).

SPARK_WORKER_WEBUI_PORT

Port?for?the?worker?web?UI?(default:?8081).

SPARK_WORKER_INSTANCES

Number?of?worker?instances?to?run?on?each?machine?(default:?1).?You?can?make?this?more?than?1?if?you?have?have?very?large?machines?and?would?like?multiple?Spark?worker?processes.?If?you?do?set?this,?make?sure?to?also?set?SPARK_WORKER_CORES?explicitly?to?limit?the?cores?per?worker,?or?else?each?worker?will?try?to?use?all?the?cores.?

SPARK_WORKER_DIR

Directory?to?run?applications?in,?which?will?include?both?logs?and?scratch?space?(default:?SPARK_HOME/work).

SPARK_WORKER_OPTS

Configuration?properties?that?apply?only?to?the?worker?in?the?form?"-Dx=y"?(default:?none).?See?below?for?a?list?of?possible?options.

SPARK_DAEMON_MEMORY

Memory?to?allocate?to?the?Spark?master?and?worker?daemons?themselves?(default:?1g).

SPARK_DAEMON_JAVA_OPTS

JVM?options?for?the?Spark?master?and?worker?daemons?themselves?in?the?form?"-Dx=y"?(default:?none).

SPARK_PUBLIC_DNS

The?public?DNS?name?of?the?Spark?master?and?workers?(default:?none).

Property?Name

Default

Meaning

spark.deploy.retainedApplications

200

The?maximum?number?of?completed?applications?to?display.?Older?applications?will?be?dropped?from?the?UI?to?maintain?this?limit.

spark.deploy.retainedDrivers

200

The?maximum?number?of?completed?drivers?to?display.?Older?drivers?will?be?dropped?from?the?UI?to?maintain?this?limit.

spark.deploy.spreadOut

true

Whether?the?standalone?cluster?manager?should?spread?applications?out?across?nodes?or?try?to?consolidate?them?onto?as?few?nodes?as?possible.?Spreading?out?is?usually?better?for?data?locality?in?HDFS,?but?consolidating?is?more?efficient?for?compute-intensive?workloads.?

spark.deploy.defaultCores

(infinite)

Default?number?of?cores?to?give?to?applications?in?Spark's?standalone?mode?if?they?don't?set?spark.cores.max.?If?not?set,?applications?always?get?all?available?cores?unless?they?configure?spark.cores.max?themselves.?Set?this?lower?on?a?shared?cluster?to?prevent?users?from?grabbing?the?whole?cluster?by?default.?

spark.worker.timeout

60

Number?of?seconds?after?which?the?standalone?deploy?master?considers?a?worker?lost?if?it?receives?no?heartbeats.?

Property?Name

Default

Meaning

spark.worker.cleanup.enabled

false

Enable?periodic?cleanup?of?worker?/?application?directories.?Note?that?this?only?affects?standalone?mode,?as?YARN?works?differently.?Only?the?directories?of?stopped?applications?are?cleaned?up.?

spark.worker.cleanup.interval

1800?(30?minutes)

Controls?the?interval,?in?seconds,?at?which?the?worker?cleans?up?old?application?work?dirs?on?the?local?machine.?

spark.worker.cleanup.appDataTtl

7?*?24?*?3600?(7?days)

The?number?of?seconds?to?retain?application?work?directories?on?each?worker.?This?is?a?Time?To?Live?and?should?depend?on?the?amount?of?available?disk?space?you?have.?Application?logs?and?jars?are?downloaded?to?each?application?work?dir.?Over?time,?the?work?dirs?can?quickly?fill?up?disk?space,?especially?if?you?run?jobs?very?frequently.?

System?property

Meaning

spark.deploy.recoveryMode

Set?to?ZOOKEEPER?to?enable?standby?Master?recovery?mode?(default:?NONE).

spark.deploy.zookeeper.url

The?ZooKeeper?cluster?url?(e.g.,?192.168.1.100:2181,192.168.1.101:2181).

spark.deploy.zookeeper.dir

The?directory?in?ZooKeeper?to?store?recovery?state?(default:?/spark).

System?property

Meaning

spark.deploy.recoveryMode

Set?to?FILESYSTEM?to?enable?single-node?recovery?mode?(default:?NONE).

spark.deploy.recoveryDirectory

The?directory?in?which?Spark?will?store?recovery?state,?accessible?from?the?Master's?perspective.