scala - ERROR while Create Delta Table with Spark-Sql

Question

I am using - Scala 2.11.8, Spark - 2.4.4, Delta - 0.4.0

My Usage =>

val deltaQuery = """CREATE TABLE <SCHEMA_NAME>.<TABLE_NAME> 
                    ( abc String, pqr binary, ingestiontime bigint) 
                      USING DELTA 
                      PARTITIONEND BY (ingestiontime)
                      LOCATION 's3a://..........'"""
 
spark.sql(deltaQuery)

ERROR - >

    Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: DELTA. Please find packages at http://spark.apache.org/third-party-projects.html
.....
.....
....(continued error stack)
....
.....
.....
    Caused by: java.lang.ClassNotFoundException: DELTA.DefaultSource
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:652)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20$$anonfun$apply$12.apply(DataSource.scala:652)
        at scala.util.Try$.apply(Try.scala:192)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:652)
        at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$20.apply(DataSource.scala:652)
        at scala.util.Try.orElse(Try.scala:84)
        at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:652)
        ... 50 more

I already have Delta Core dependency in POM.

        <dependency>
            <groupId>io.delta</groupId>
            <artifactId>delta-core_${scala.binary.version}</artifactId>
            <version>0.4.0</version>
            <scope>provided</scope>
        </dependency>

Thanks in Advance.

FYI -

1. I have already been able to run such queries via Databricks (Without having delta core in dependency) using the Databricks Runtime consisting of Databricks Delta

2. On AWS EMR , I am facing the above issue even after adding delta core dependency in POM.xml.

question from:https://stackoverflow.com/questions/66047545/error-while-create-delta-table-with-spark-sql

Answer 1

Waitting for answers