Tmap between hadoop tables and Sql servers

One Star

Tmap between hadoop tables and Sql servers

I am comparing(Data validation row by row comparison)  two tables in hadoop database and SQL servers using tmap component.
The HDFS file size - 30.3 GB and Record Count is - 328268683.
Error is as below :- There is not enough space in hard disk.
Configuration - 64 bit ,8 GB RAM,64GB hard disk.
Please let me know incase of any other alternatives other than tmap.
Erro log
Starting job ImpalaDemoFA_SS_BarcodeTransaction_QA at 11:30 18/06/2015.





connecting to socket on port 3623

connected

java.lang.reflect.InvocationTargetException

at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithMethod(RegularObjectPersister.java:120)

at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:86)

at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

at org.jboss.serial.io.JBossObjectOutputStream.writeObjectOverride(JBossObjectOutputStream.java:181)

at java.io.ObjectOutputStream.writeObject(Unknown Source)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA$row2Struct.writeValuesData(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5317)

: org.jboss.serial.persister.RegularObjectPersister - error


java.lang.reflect.InvocationTargetException

at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithMethod(RegularObjectPersister.java:120)

at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:86)

at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

at org.jboss.serial.io.JBossObjectOutputStream.writeObjectOverride(JBossObjectOutputStream.java:181)

at java.io.ObjectOutputStream.writeObject(Unknown Source)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA$row2Struct.writeValuesData(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5317)

at org.talend.designer.components.lookup.persistent.PersistentSortedLookupManager.writeBuffer(PersistentSortedLookupManager.java:417)

at org.talend.designer.components.lookup.persistent.PersistentSortedLookupManager.put(PersistentSortedLookupManager.java:236)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.tJDBCInput_1Process(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5778)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.tMSSqlInput_1Process(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:2213)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.runJobInTOS(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:6070)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.main(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5927)

Caused by: java.io.IOException: There is not enough space on the disk

at java.io.FileOutputStream.writeBytes(Native Method)


at org.talend.designer.components.lookup.persistent.PersistentSortedLookupManager.writeBuffer(PersistentSortedLookupManager.java:417)

at org.talend.designer.components.lookup.persistent.PersistentSortedLookupManager.put(PersistentSortedLookupManager.java:236)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.tJDBCInput_1Process(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5778)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.tMSSqlInput_1Process(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:2213)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.runJobInTOS(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:6070)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.main(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5927)

Caused by: java.io.IOException: There is not enough space on the disk

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(Unknown Source)

at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

at java.io.BufferedOutputStream.write(Unknown Source)

at java.io.DataOutputStream.writeLong(Unknown Source)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeLong(DataContainer.java:254)

at org.jboss.serial.classmetamodel.StreamingClass.saveStream(StreamingClass.java:64)

at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:265)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)

at org.jboss.serial.persister.ObjectOutputStreamProxy.writeFields(ObjectOutputStreamProxy.java:79)

at org.jboss.serial.persister.ObjectOutputStreamProxy.defaultWriteObject(ObjectOutputStreamProxy.java:68)

at java.math.BigDecimal.writeObject(Unknown Source)

... 17 more


at java.io.FileOutputStream.write(Unknown Source)

at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

at java.io.BufferedOutputStream.write(Unknown Source)

at java.io.DataOutputStream.writeLong(Unknown Source)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeLong(DataContainer.java:254)

at org.jboss.serial.classmetamodel.StreamingClass.saveStream(StreamingClass.java:64)

at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:265)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)

at org.jboss.serial.persister.ObjectOutputStreamProxy.writeFields(ObjectOutputStreamProxy.java:79)

at org.jboss.serial.persister.ObjectOutputStreamProxy.defaultWriteObject(ObjectOutputStreamProxy.java:68)

at java.math.BigDecimal.writeObject(Unknown Source)

... 17 more

Exception in component tAdvancedHash_row2

java.lang.RuntimeException: java.io.IOException

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA$row2Struct.writeValuesData(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5320)

at org.talend.designer.components.lookup.persistent.PersistentSortedLookupManager.writeBuffer(PersistentSortedLookupManager.java:417)

at org.talend.designer.components.lookup.persistent.PersistentSortedLookupManager.put(PersistentSortedLookupManager.java:236)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.tJDBCInput_1Process(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5778)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.tMSSqlInput_1Process(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:2213)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.runJobInTOS(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:6070)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA.main(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5927)


disconnected

Caused by: java.io.IOException

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithMethod(RegularObjectPersister.java:129)

at org.jboss.serial.persister.RegularObjectPersister.defaultWrite(RegularObjectPersister.java:86)

at org.jboss.serial.persister.RegularObjectPersister.writeData(RegularObjectPersister.java:62)

at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:276)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

at org.jboss.serial.io.JBossObjectOutputStream.writeObjectOverride(JBossObjectOutputStream.java:181)

at java.io.ObjectOutputStream.writeObject(Unknown Source)

at music_connect.impalademofa_ss_barcodetransaction_qa_0_1.ImpalaDemoFA_SS_BarcodeTransaction_QA$row2Struct.writeValuesData(ImpalaDemoFA_SS_BarcodeTransaction_QA.java:5317)

... 6 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithMethod(RegularObjectPersister.java:120)

... 13 more

Caused by: java.io.IOException: There is not enough space on the disk

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(Unknown Source)

at java.io.BufferedOutputStream.flushBuffer(Unknown Source)

at java.io.BufferedOutputStream.write(Unknown Source)

at java.io.DataOutputStream.writeLong(Unknown Source)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeLong(DataContainer.java:254)

at org.jboss.serial.classmetamodel.StreamingClass.saveStream(StreamingClass.java:64)

at org.jboss.serial.objectmetamodel.ObjectDescriptorFactory.describeObject(ObjectDescriptorFactory.java:265)

at org.jboss.serial.objectmetamodel.DataContainer$DataContainerDirectOutput.writeObject(DataContainer.java:206)

at org.jboss.serial.persister.RegularObjectPersister.writeSlotWithFields(RegularObjectPersister.java:182)

at org.jboss.serial.persister.ObjectOutputStreamProxy.writeFields(ObjectOutputStreamProxy.java:79)

at org.jboss.serial.persister.ObjectOutputStreamProxy.defaultWriteObject(ObjectOutputStreamProxy.java:68)

at java.math.BigDecimal.writeObject(Unknown Source)

... 17 more


Job ImpalaDemoFA_SS_BarcodeTransaction_QA ended at 13:53 18/06/2015.

Regards,
Raakesh R
Moderator

Re: Tmap between hadoop tables and Sql servers

Hi,
Have you already stored temporary data on disk?
What's kind of comparison? Could you please give us more description about your job requirement so that we can see if there is any other alternatives.

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.