Jmeter - Hive giving java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer

Question

I am using Jmeter to create load on Hive.

While running the Jmeter I am getting the follwing error in the file jmeter.log :

 2021-02-05 10:11:02,927 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[HiveThreadGroupTWANG 1-1,5,main]
 java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge23.<clinit>(HadoopThriftAuthBridge23.java:66) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.getBridge(HadoopThriftAuthBridge.java:82) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]

at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
 Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.SaslRpcServer
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
    ... 21 more

The extra jars that I have added in the lib/ext directory are: old-hive-exec-3.1.0.3.1.5.0-152.jar hadoop-common-3.1.1.3.1.5.0-152.jar.bkp slf4j-log4j12-1.7.25.jar slf4j-api-1.7.25.jar old-hive-service-3.1.0.3.1.5.0-152.jar old-hive-metastore-3.1.0.3.1.5.0-152.jar old-hive-jdbc-handler-3.1.0.3.1.5.0-152.jar old-hive-jdbc-3.1.0.3.1.5.0-152-standalone.jar.bkp old-hive-jdbc-3.1.0.3.1.5.0-152.jar log4j-1.2.17.jar libthrift-0.9.3-1.jar libfb303-0.9.3.jar hive-service-3.1.2.jar hive-metastore-3.1.2.jar hive-jdbc-3.1.2.jar hive-exec-3.1.2.jar curator-client-5.1.0.jar curator-framework-5.1.0.jar zookeeper-3.6.2.jar zookeeper-3.3.3.jar hadoop-common-3.1.1.3.1.0.0-78.jar.bkp

After moving hadoop-common-3.1.1.3.1.5.0-152.jar to the lib directory I am getting the follwoing error.

  java.lang.NoClassDefFoundError: org/apache/commons/configuration2/Configuration
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:43) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:41) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:143) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:276) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
    at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:95) ~[hive-exec-3.1.2.jar:3.1.2]
    at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
    at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]

      at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]
    at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
 Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.Configuration
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
    ... 24 more

question from:https://stackoverflow.com/questions/66062062/jmeter-hive-giving-java-lang-noclassdeffounderror-org-apache-hadoop-security

Answer 1

1. You need to add hadoop-common .jar file to JMeter Classpath
2. Dependency libraries should go for lib folder, not to lib/ext folder, lib/ext is for JMeter Plugins
3. You have hadoop-common-3.1.1.3.1.5.0-152.jar.bkp, I don't know what is .bkp file but I believe it should have .jar extension, most probably you downloaded something else
4. Are you aware of jmeter-hadoop plugin? It looks like it can be useful for your use case, at least you will get dependency jars managed. See How to Load Test HBase with JMeter article for more details