I was trying to convert Informatica mappings to Talend.
Following table stats
1. Lookup-1: 28M (PostgreSQL Input with SQL join & Filter) - Cursor Size - 1M
2. Lookup-2: 35M (PostgreSQL Input with SQL Filter) - Cursor Size - 1M
Lookups in Parallel
3. Main Table: 27M (PostgreSQL Joins with multiple tables and Date Filters)
Max Memory Settings I provided was 8GB
Lookups are running fine, but once it reaches Main Table read it will slowdowns and after 1hr of running the whole process it comes out as Java Heap Memory error.
Not sure what else I got to look to make this work. And this is just one small Mapping and going fwd down the line got too complex mappings with huge data.
Does parallelization help?
Does Multi Thread execution help? If Yes, what buffer unit size should set to?
Or Custom Batch processing process every 5M records will help?
Please do advice, Thanks.
Tag as "solved" for others! Kudos to thanks!
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema