apache spark - How to change job/stage description in web UI?

Question

When I run a job on Apache Spark, the web UI gives a view similar to this:

While this is incredibly useful for me as a developer to see where things are, I think the line numbers in the stage description would be not quite as useful for my support team. To make their job easier, I would like to have the ability to provide a bespoke name for each stage of my job, as well as for the job itself, like so:

Is this something that can be done in Spark? If so, how would I do so?

Answer 1

That's where one of the very uncommon features of Spark Core called local properties applies so well.

Spark SQL uses it to group different Spark jobs under a single structured query so you can use SQL tab and navigate easily.

You can control local properties using SparkContext.setLocalProperty:

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. User-defined properties may also be set here. These properties are propagated through to worker tasks and can be accessed there via org.apache.spark.TaskContext#getLocalProperty.

web UI uses two local properties:

• callSite.short in Jobs tab (and is exactly what you want)
• callSite.long in Job Details page.

Sample Usage

scala> sc.setLocalProperty("callSite.short", "callSite.short")

scala> sc.setLocalProperty("callSite.long", "this is callSite.long")

scala> sc.parallelize(0 to 9).count
res2: Long = 10

And the result in web UI.

Click a job to see the details where you can find the longer call site, i.e. callSite.long.

Here comes the Stages tab.