Error in batch job writing to Hive

Five Stars

Error in batch job writing to Hive

Hello all,

 

I am trying to insert some rows in a table in Hive using big data batch job(map reduce)  with tJDBCOutput.2.PNGjob.PNG

However, I am facing the following error:

Running job: job_1527425158191_0398
map 0% reduce 0%
Task Id : attempt_1527425158191_0398_m_000000_0, Status : FAILED

Error: java.lang.ClassNotFoundException: org.apache.hive.service.cli.thrift.TCLIService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
Task Id : attempt_1527425158191_0398_m_000001_0, Status : FAILED
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Task Id : attempt_1527425158191_0398_m_000000_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Task Id : attempt_1527425158191_0398_m_000001_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Error: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
Task Id : attempt_1527425158191_0398_m_000000_2, Status : FAILED
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Task Id : attempt_1527425158191_0398_m_000001_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.apache.hive.service.cli.thrift.TCLIService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at testproj.testmr_0_1.testmr$tJDBCOutput_1StructOutputFormat.getRecordWriter(testmr.java:864)
at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:836)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:447)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
map 100% reduce 0%
Job complete: job_1527425158191_0398
Counters: 13
Job Counters
Failed map tasks=7
Killed map tasks=1
Launched map tasks=8
Other local map tasks=6
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=23421
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=23421
Total vcore-milliseconds taken by all map tasks=23421
Total megabyte-milliseconds taken by all map tasks=23983104
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Job Failed: Task failed task_1527425158191_0398_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

java.io.IOException: Job failed!
at org.talend.hadoop.mapred.lib.MRJobClient.runJob(MRJobClient.java:154)
at testproj.testmr_0_1.testmr.runMRJob(testmr.java:1782)
at testproj.testmr_0_1.testmr.access$1(testmr.java:1772)
at testproj.testmr_0_1.testmr$1.run(testmr.java:1606)
at testproj.testmr_0_1.testmr$1.run(testmr.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at testproj.testmr_0_1.testmr.tFileInputDelimited_1_HDFSInputFormatProcess(testmr.java:1540)
at testproj.testmr_0_1.testmr.tLibraryLoad_1Process(testmr.java:599)
at testproj.testmr_0_1.testmr.run(testmr.java:1750)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at testproj.testmr_0_1.testmr.runJobInTOS(testmr.java:1705)
at testproj.testmr_0_1.testmr.main(testmr.java:1653)
[statistics] disconnected

Five Stars

Re: Error in batch job writing to Hive

Hey guys, I was able to fix this issue - all those jars had to be added :


Capture.PNG

But then I got another error: Method dbrecordwriter.write not supported. Here some guys are experiencing a similar error: https://stackoverflow.com/questions/34067686/spark-1-5-1-not-working-with-hive-jdbc-1-2-0 

It is explicitly not implemented in the hive jdbc library branch-1.2 .