Mutiple Masters for Spark and Zookeeper HA

I am trying to setup multi master spark for High Availability in Standalone mode.

I have 6 node cluster: 6 workers and 2 masters are available.

All of the 6 workers are showing up on Master 1 UI, where as spark master 2 doesnt have any workers listed? Do i need to start the workers on master 2 also?

Or will ZooKeeper switch those worked to master 2 in case of failure of master 1?

One more thing is i have ZooKeeper setup for kafka on same nodes. Now do i need to install ZooKeeper separately for Spark? Does the new ZooKeeper needs to have different port number from Kafkas ZooKeeper?

Copyright Notice：Content Author:「AKC」，Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article：https://stackoverflow.com/questions/39936645/mutiple-masters-for-spark-and-zookeeper-ha

Products recommended by Amazon

More >>>

SanDisk 128GB microSDXC-Card, Licensed for Nintendo-Switch - SDSQXAO-128G-GNCZN

HUANUO Dual Monitor Stand - Adjustable Spring Monitor Desk Mount Swivel Vesa Bracket with C Clamp, Grommet Mounting Base for 13 to 27 Inch Computer Screens - Each Arm Holds 4.4 to 14.3lbs

AMD Ryzen 5 5600X 6-core, 12-Thread Unlocked Desktop Processor with Wraith Stealth Cooler

Timberland Men's 6-Inch Premium Waterproof Boot

More about “Mutiple Masters for Spark and Zookeeper HA” related questions

Mutiple Masters for Spark and Zookeeper HA

I am trying to setup multi master spark for High Availability in Standalone mode. I have 6 node cluster: 6 workers and 2 masters are available. All of the 6 workers are showing up on Master 1 UI,

Show Detail

Spark standalone master HA jobs in WAITING status

We are trying to setup HA on spark standalone master using zookeeper. We have two zookeeper hosts which we are using for spark ha as well. Configured following thing in spark-env.sh

Show Detail

HA setup of Spark Cluster using Zookeeper Quorum

We are trying to setup Spark HA setup with ZK. We have 2 machines for Master for Spark process and another 3 for Spark Slaves The configuration In Master Machine for spark HA done as below in spar...

Show Detail

How to integrate Zookeeper with Spark Driver program?

Trying to integrate Spark Standalone cluster with Zookeeper And I have a question on integration from Spark Driver program perspective. I have 3 Spark Masters and I understand one of them will be

Show Detail

How to set up Spark with Zookeeper for HA?

I want to config Apache spark master to connect with Zookeeper I have installed both of them and run Zookeeper. In spark-env.sh, I add 2 lines: -Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.dep...

Show Detail

Active master not accepting new applications if one of the masters added to zookeeper is down

I am facing a very weird issue while enabling high availability(HA) in spark stand alone cluster. I have configured 3 spark masters and registered them in zookeeper by following below steps: Crea...

Show Detail

Flink 1.2 does not start in HA Cluster mode

I've installed Flink 1.2 in HA cluster mode 2 JobManagers 1 TaskManager locally and it kept refusing to actually start in this mode showing "Starting cluster." message instead of "Starting HA clus...

Show Detail

Statefulset Spark Master HA in Kubernetes If Kubernetes node running master pod itself goes down

I have deployed Spark on kubernetes with 1 master and 3 worker replicas (statefulset). I know that by virtue of K8s we get HA i.e. If Spark Master Pod goes down, K8s can restart the master pod. In ...

Show Detail

How to specify multiple Spark Standalone masters (for spark.master property)?

I have 1 master with 3 worker nodes communicating to master. As a disaster recovery we have created 2 Masters and let zookeeper elect the master. I am using datastax's spark Cassandra connector. Is

Show Detail

Why does Apache Spark Master shutdown when Zookeeper expires the session

As I understand, Apache Spark Master can be run in high availability mode using Zookeeper. That is, multiple Spark masters can run in Leader/Follower mode and these modes are registered with ZooKee...

Show Detail