Thursday, April 26, 2018

java - Unable to start worker node for Apache Spark 2.1.1




I am unable to start worker node for Apache Spark 2.1.1, (I have single node setup, my Master and Worker will be running on same host), below are the logs:



$ ./start-all.sh



starting org.apache.spark.deploy.master.Master,
logging to

/home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.master.Master-1-myhost.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to
/home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out
localhost: failed to launch: nice -n 0
/home/kedar/software/spark/bin/spark-class
org.apache.spark.deploy.worker.Worker --webui-port 8081
spark://myhost:7077 localhost: at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) localhost: at
sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
localhost: full log in

/home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out




below are logs:



$ cat /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out



Exception in thread "main" java.lang.UnsupportedClassVersionError:

org/apache/spark/launcher/Main : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:803) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:442) at
java.net.URLClassLoader.access$100(URLClassLoader.java:64) at
java.net.URLClassLoader$1.run(URLClassLoader.java:354) at
java.net.URLClassLoader$1.run(URLClassLoader.java:348) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:347) at

java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)




Any help is appreciated! Thanks!


Answer



The issue was my java alternatives were pointing to wrong java version, below are the steps to resolve the issue on ubuntu, and as we could see the Spark Also started successfully:




kedar@myhost:/opt$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/opt/jdk1.8.0_72/bin/javac"
javac
kedar@myhost:/opt$ sudo update-alternatives --install "/usr/bin/javac" "javac" "/opt/jdk1.8.0_72/bin/javac" 1
kedar@myhost:/opt$ ll javac
ls: cannot access javac: No such file or directory
kedar@myhost:/opt$ sudo update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
Nothing to configure.
kedar@myhost:/opt$ sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).


Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-7-openjdk-amd64/bin/javac 1071 auto mode
1 /opt/jdk1.8.0_72/bin/javac 1 manual mode
2 /usr/lib/jvm/java-7-openjdk-amd64/bin/javac 1071 manual mode

Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /opt/jdk1.8.0_72/bin/javac to provide /usr/bin/javac (javac) in manual mode
kedar@myhost:/opt$ sudo update-alternatives --config javac

There are 2 choices for the alternative javac (providing /usr/bin/javac).

Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-7-openjdk-amd64/bin/javac 1071 auto mode
* 1 /opt/jdk1.8.0_72/bin/javac 1 manual mode
2 /usr/lib/jvm/java-7-openjdk-amd64/bin/javac 1071 manual mode

Press enter to keep the current choice[*], or type selection number: kedar@myhost:/opt$
kedar@myhost:/opt$

kedar@myhost:/opt$
kedar@myhost:/opt$ sudo update-alternatives --install "/usr/bin/java" "java" "/opt/jdk1.8.0_72/bin/java" 1
kedar@myhost:/opt$ ll javac
ls: cannot access javac: No such file or directory
kedar@myhost:/opt$ sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1071 auto mode

1 /opt/jdk1.8.0_72/bin/java 1 manual mode
2 /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java 1071 manual mode

Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /opt/jdk1.8.0_72/bin/java to provide /usr/bin/java (java) in manual mode
kedar@myhost:/opt$ cd
kedar@myhost:~$ cd $SPARK_HOME
kedar@myhost:~/software/spark$ ./sbin/stop-all.sh
localhost: no org.apache.spark.deploy.worker.Worker to stop
stopping org.apache.spark.deploy.master.Master

kedar@myhost:~/software/spark$ ./sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.master.Master-1-myhost.out
localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/kedar/software/spark/logs/spark-kedar-org.apache.spark.deploy.worker.Worker-1-myhost.out

No comments:

Post a Comment

plot explanation - Why did Peaches' mom hang on the tree? - Movies & TV

In the middle of the movie Ice Age: Continental Drift Peaches' mom asked Peaches to go to sleep. Then, she hung on the tree. This parti...