Error when deduping in Talend Data Integration 6.2.1 in Critical Job

Highlighted
Four Stars

Error when deduping in Talend Data Integration 6.2.1 in Critical Job

Hi there, looking for help ASAP.  I'm getting this error when trying to dedup a HUGE text pipe delimited file.  It is about 120GB.  When I get this error, I usually play with the JVM arguments and I get it to run, but in this case I am not able to remedy it so far with changing the JVM arguments.  It processes to about the 50M row level and it has over 100 million rows.

 

The latest arguments I had were this:

https://www.screencast.com/t/Sr0GU4B8kq9

 

And this is my workflow which is pretty simple.  I just need to dedup the GetCurrentFile.

https://www.screencast.com/t/YZ7wYsI8BXJ

 

Any thoughts?

Thank you

 

 

 

Moderator

Re: Error when deduping in Talend Data Integration 6.2.1 in Critical Job

Hello,

Could you please show us the error full stack trace?

Please feel free to let us know if these related articles help.

https://community.talend.com/t5/Migration-Configuration-and/OutOfMemory-Exception/ta-p/21669

https://community.talend.com/t5/Design-and-Development/Could-not-reserve-enough-space-for-object-hea...

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog