mardi 4 août 2015

Apache Spark Sql issue in multi node hadoop cluster

Hi I am using Spark java apis to fetch data from hive. This code is working in hadoop single node cluster. But when I tried to use it in hadoop multi node cluster it throws error as

org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.

Note : I have used master as local for single node and yarn-cluster for multi node.

And this is my java code

 SparkConf sparkConf = new SparkConf().setAppName("Hive").setMaster("yarn-cluster");
 JavaSparkContext ctx = new JavaSparkContext(sparkConf);
 HiveContext sqlContext = new HiveContext(ctx.sc());
org.apache.spark.sql.Row[] result = sqlContext.sql("Select * from Tablename").collect();

Also I have tried to change master as local and now it throws unknown hostname exception.
Can anyone help me in this?



via Chebli Mohamed

Aucun commentaire:

Enregistrer un commentaire