spark-submit on local Hadoop-Yarn setup, fails with Stdout path must be absolute error

Question

I have installed the latest Hadoop and Spark versions on my Windows machine. I am trying to launch one of the provided examples but it fails and I have no idea what the diagnostic means. It seems it's related to the stdout but I can't figure out the root cause.

I launch the following command:

spark-submit --master yarn --class org.apache.spark.examples.JavaSparkPi C:spark-3.0.1-bin-hadoop3.2examplesjarsspark-examples_2.12-3.0.1.jar 100

And the exception I have is:

21/01/25 10:53:53 WARN MetricsSystem: Stopping a MetricsSystem that is not running 21/01/25 10:53:53 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 21/01/25 10:53:53 INFO SparkContext: Successfully stopped SparkContext Exception in thread "main" org.apache.spark.SparkException: Application application_1611568137841_0002 failed 2 times due to AM Container for appattempt_1611568137841_0002_000002 exited with exitCode: -1 Failing this attempt.Diagnostics: [2021-01-25 10:53:53.381] Stdout path must be absolute

For more detailed output, check the application tracking page: http://xxxx-PC:8088/cluster/app/application_1611568137841_0002 Then click on links to logs of each attempt. . Failing the application. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBack end.scala:95) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:201) at org.apache.spark.SparkContext.(SparkContext.scala:555) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928) at org.apache.spark.examples.JavaSparkPi.main(JavaSparkPi.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 21/01/25 10:53:53 INFO ShutdownHookManager: Shutdown hook called 21/01/25 10:53:53 INFO ShutdownHookManager: Deleting directory C:UsersxxxAppDataLocalTempspark-b28ecb32-5e3f-4d6a-973a-c03a7aae0da9 21/01/25 10:53:53 INFO ShutdownHookManager: Deleting directory C:Users/xxxAppDataLocalTempspark-3665ba77-d2aa-424a-9f75-e772bb5b9104

As for the diagnostics:

Diagnostics:
Application application_1611562870926_0004 failed 2 times due to AM Container for appattempt_1611562870926_0004_000002 exited with exitCode: -1 Failing this attempt.Diagnostics: [2021-01-25 10:29:19.734]Stdout path must be absolute For more detailed output, check the application tracking page: http://****-PC:8088/cluster/app/application_1611562870926_0004 Then click on links to logs of each attempt. . Failing the application.

Thank you !

question from:https://stackoverflow.com/questions/65882951/spark-submit-on-local-hadoop-yarn-setup-fails-with-stdout-path-must-be-absolute

Answer 1

So I am not sure of the root cause yet, it's probably due to the fact that I run under windows and some default property was wrong for Yarn. When I added the 2 following properties on yarn-site.xml, it worked fine:

   <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/tmp</value>
   </property>
   <property>
    <name>yarn.log.dir</name>
    <value>/tmp</value>
   </property>

Hope it helps someone in the future !