I have a requirement to upload rows from a CSV file with a delay in between each row to allow batch processing to take place. I am running one row at a time as the logic behind the application is complex and, to stop 'too many SQL statements' exceptions, a batch size of 1 is to be used.
My question is' How can I setup Talend to do this?
I have a tFileInputDelimited linked to a tSleep then linked to a tSalesforceOutput component. I have set the attribute "Rows to Commit" to be 1 and so I have 2 transactions in Salesforce; one for each record in my test data. I am trying use the tSleep to put a 20 second pause between each call to Salesforce. The result is that the 2 records are created in seperate transactions but updated with no delay. What other components/configuration do I need?
Any help would be appreciated.
I had the batch setting to 1 and so was working correctly - I just needed to insert a tJavaRow component and add the line Thread.sleep(integer) to pause between each update.
So now I have a tFileInputDelimited linked to tJavaRow then to tSalesforceOuput. It all works as expected now.
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn how and why companies are moving to the Cloud
Accelerate your data lake projects with an agile approach
Create systems and workflow to manage clean data ingestion and data transformation.