scala - How can I get the current SparkSession in any place of the codes?

Question

I have created a session in the main() function, like this:

val sparkSession = SparkSession.builder.master("local[*]").appName("Simple Application").getOrCreate()

Now if I want to configure the application or access the properties, I can use the local variable sparkSession in the same function.

What if I want to access this sparkSession elsewhere in the same project, like project/module/.../.../xxx.scala. What should I do?

Answer 1

Once a session was created (anywhere), you can safely use:

SparkSession.builder.getOrCreate()

To get the (same) session anywhere in the code, as long as the session is still alive. Spark maintains a single active session so unless it was stopped or crashed, you'll get the same one.

Edit: builder is not callable, as mentioned in the comments.