Hi I am using Spark java apis to fetch data from hive. This code is working in hadoop single node cluster. But when I tried to use it in hadoop multi node cluster it throws error as
org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.
Note : I have used master as local for single node and yarn-cluster for multi node.
And this is my java code
SparkConf sparkConf = new SparkConf().setAppName("Hive").setMaster("yarn-cluster");
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
HiveContext sqlContext = new HiveContext(ctx.sc());
org.apache.spark.sql.Row[] result = sqlContext.sql("Select * from Tablename").collect();
Also I have tried to change master as local and now it throws unknown hostname exception.
Can anyone help me in this?
via Chebli Mohamed
Aucun commentaire:
Enregistrer un commentaire