Hi there, looking for help ASAP. I'm getting this error when trying to dedup a HUGE text pipe delimited file. It is about 120GB. When I get this error, I usually play with the JVM arguments and I get it to run, but in this case I am not able to remedy it so far with changing the JVM arguments. It processes to about the 50M row level and it has over 100 million rows.
The latest arguments I had were this:
And this is my workflow which is pretty simple. I just need to dedup the GetCurrentFile.
Could you please show us the error full stack trace?
Please feel free to let us know if these related articles help.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema
Pick up some tips and tricks with Context Variables