apache spark - Databricks-connect Java connects to local instead of remote

Question

I have a Java application that connects to an Apache Spark cluster and performs some operations. I'm trying to connect to a Databricks cluster on Azure, using databricks-connect 7.3. If I run from the terminal databricks-connect test, everything works perfectly. I'm following their documentation, I included the jars in IntelliJ, added spark.databricks.service.server.enabled true to the cluster in Databricks and used the following to create the SparkSession:

SparkSession spark = SparkSession
                .builder()
                .master("local")
                .getOrCreate();

The problem is that this command connects to a local cluster that is instantiated at runtime, and does not connect to the remote Databricks cluster. Am I missing something?

question from:https://stackoverflow.com/questions/66048649/databricks-connect-java-connects-to-local-instead-of-remote

Answer 1

Waitting for answers