Error in batch job writing to Hive

Highlighted
Five Stars

Error in batch job writing to Hive

Hello all,

 

I am trying to insert some rows in a table in Hive using big data batch job(map reduce)  with tJDBCOutput.2.PNGjob.PNG

However, I am facing the following error:

Running job: job_1527425158191_0398
map 0% reduce 0%
Task Id : attempt_1527425158191_0398_m_000000_0, Status : FAILED

Error: java.lang.ClassNotFoundException: org.apache.hive.service.cli.thrift.TCLIService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
Task Id : attempt_1527425158191_0398_m_000001_0, Status : FAILED
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Task Id : attempt_1527425158191_0398_m_000000_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Task Id : attempt_1527425158191_0398_m_000001_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
Task Id : attempt_1527425158191_0398_m_000000_2, Status : FAILED
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Task Id : attempt_1527425158191_0398_m_000001_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.hive.service.cli.thrift.TCLIService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
map 100% reduce 0%
Job complete: job_1527425158191_0398
Counters: 13
Job Counters
Failed map tasks=7
Killed map tasks=1
Launched map tasks=8
Other local map tasks=6
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=23421
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=23421
Total vcore-milliseconds taken by all map tasks=23421
Total megabyte-milliseconds taken by all map tasks=23983104
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Job Failed: Task failed task_1527425158191_0398_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

java.io.IOException: Job failed!
at org.talend.hadoop.mapred.lib.MRJobClient.runJob(MRJobClient.java:154)
at testproj.testmr_0_1.testmr.runMRJob(testmr.java:1782)
at testproj.testmr_0_1.testmr.access$1(testmr.java:1772)
at testproj.testmr_0_1.testmr$1.run(testmr.java:1606)
at testproj.testmr_0_1.testmr$1.run(testmr.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at testproj.testmr_0_1.testmr.tFileInputDelimited_1_HDFSInputFormatProcess(testmr.java:1540)
at testproj.testmr_0_1.testmr.tLibraryLoad_1Process(testmr.java:599)
at testproj.testmr_0_1.testmr.run(testmr.java:1750)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at testproj.testmr_0_1.testmr.runJobInTOS(testmr.java:1705)
at testproj.testmr_0_1.testmr.main(testmr.java:1653)
[statistics] disconnected

Five Stars

Re: Error in batch job writing to Hive

Hey guys, I was able to fix this issue - all those jars had to be added :


Capture.PNG

But then I got another error: Method dbrecordwriter.write not supported. Here some guys are experiencing a similar error: https://stackoverflow.com/questions/34067686/spark-1-5-1-not-working-with-hive-jdbc-1-2-0 

It is explicitly not implemented in the hive jdbc library branch-1.2 .

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now