hadoop yarn - How to exit spark-submit after the submission

Question

When submitting spark streaming program using spark-submit(YARN mode) it keep polling the status and never exit

Is there any option in spark-submit to exit after the submission?

===why this trouble me===

The streaming program will run forever and i don't need the status update

I can ctrl+c to stop it if i start it manually but i have lots of streaming context to start and i need to start them using script

I can put the spark-submit program in background, but after lots of background java process created, the user corresponding to, will not able to run any other java process because JVM cannot create GC thread

Answer 1

I know this is an old question but there's a way to do this now by setting --conf spark.yarn.submit.waitAppCompletion=false when you're using spark-submit. With this the client will exit after successfully submitting the application.

In YARN cluster mode, controls whether the client waits to exit until the application completes. If set to true, the client process will stay alive reporting the application's status. Otherwise, the client process will exit after submission.

Also, you may need to set --deploy-mode to cluster

In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application.

More at https://spark.apache.org/docs/latest/running-on-yarn.html