One Star

fastest way for talend to update multi million record

hi,
i have multi million records inside table and i wants to create talend job to update any records in the table with fastest way to load it.
is there any way? please help me Smiley Sad

regards,
kal
7 REPLIES
Four Stars

Re: fastest way for talend to update multi million record

Hi,
Can you please elaborate the case?
Which database ? Whether you have indexes on the table? how many identifiable keys are there? Do you have any transformations? etc...
Vaibhav
One Star

Re: fastest way for talend to update multi million record

hi,
i have more than 50 million records in table A (mssql). identifiable key was national id and dont have any complex transformation. eventually the update is for example to update marital status, addresses.
by having more than 50 million records inside table A, talend need to read row-by-row to search the respective national id then update either marital status or addresses.

kal
Four Stars

Re: fastest way for talend to update multi million record

What is your expectation of fastest? how many records update to complete in how much duration?
Speed also depends upon your machine's memory and cpu as well.
What is your current design? tMap would be handy to do this type of work.
What is the source of reference data to compare with Table A?
Is this process one time or recurring?
Do you have time stamp in source data to identify the updated records?
Vaibhav
One Star

Re: fastest way for talend to update multi million record

hi vaibhav,
my data approximately around 50 million. i wants to update the record within 30min and this is recurring process. also, i dont have any time stamp in source data. it based on user id.

kal
Four Stars

Re: fastest way for talend to update multi million record

Hi,
Can you experiment with ELT component for your database? Pl refer to
https://help.talend.com/search/all?query=tELTMSSqlOutput&content-lang=en
http://bekwam.blogspot.in/2011/10/updating-table-with-talend-open-studio.html
https://help.talend.com/search/all?query=tELTOracleOutput&content-lang=en
Once done, let know the performance and we will try to optimize it.
Vaibhav
One Star

Re: fastest way for talend to update multi million record

hi,
I have try create the job but keep hits error nullpointers.
Four Stars

Re: fastest way for talend to update multi million record

What is the flow of your job? can you put exact error and screenshot of your job?
Vaibhav