apache spark - Sparksql failed to execute hive lzo compressed external table

Question

I use spark SQL to load the hive table with LZO compression. The error is as follows：

Caused by: java.lang.ClassNotFoundException: com.hadoop.mapred.DeprecatedLzoTextInputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:239) at org.apache.spark.sql.hive.client.HiveClientImpl$.org$apache$spark$sql$hive$client$HiveClientImpl$$toInputFormat(HiveClientImpl.scala:892) at

What I have done:

set 
spark.driver.extraClassPath /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.3.1.4.0-315.jar
spark.driver.extraLibraryPath /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64

question from:https://stackoverflow.com/questions/65897099/sparksql-failed-to-execute-hive-lzo-compressed-external-table

Answer 1

Waitting for answers