Solved! Go to Solution.
How many records are in your main recordset?
Can you show us a screenshot of your job and your tMap configuration?
Is it possible for you to load the Excel data into a database first?
Here are screenshots below
I will give you an overview of my task I have an excel file with four excel sheets in it.Each excel sheet will be moved into four different tables in same database,which has at least Fifty thousand to 1 million this updates every week.Now the first load of excel file is done now if the second excel file comes after one week i should also find duplicates so i used lookup to catch it but checking 1 million records is a very big issue
Check the screenshots
Well first of all I would decide whether you need to have all of the processing in one job? Can the processing be split into multiple jobs? Does it all need to be done at the same time? The next thing you need to do is decide how you are joining. As I said in my last post, it doesn't look like you have any joins in your tMap. This will not work as what you are doing is essentially multiplying the rows of the main input by the number of rows in the lookup. You need to decide how you can join the data (what columns you need to match) and then move on from there.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Part 2 of a series on Context Variables
Learn how to do cool things with Context Variables
Find out how to migrate from one database to another using the Dynamic schema