Loading only correct data from flat file to azure SQL Data warehouse and storing rejected data into the flat file

Highlighted
Four Stars

Loading only correct data from flat file to azure SQL Data warehouse and storing rejected data into the flat file

Hi, I am trying to load correct data from flat file to Azure SQL data warehouse and want to capture the rejected records into the target flat file.

In my case,only erroneous records are getting captured but correct records are not getting loaded into the azure SQL data warehouse table.

I have unchecked the "Die on error" and "Use Batch Size" options

(In schema configuration of azure SQL data warehouse table,I have put less length for Full_Name than actual. Hence, 6 records should be loaded into azure SQL DW table and 14 records should get rejected)

But in my case, correct records are not getting loaded.

 

 

 

Employee

Re: Loading only correct data from flat file to azure SQL Data warehouse and storing rejected data into the flat file

Hi,

 

   The reason most probably is that you have not done committing of the data. Please add a tSQLDWHConnection component and select auto commit in advanced options.

image.png

 

Another option is to connect a tSQLDWHCOmmit component as OnSubJobOk to previous step.

 

Either way, once you commit the records, it will be available in database.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Four Stars

Re: Loading only correct data from flat file to azure SQL Data warehouse and storing rejected data into the flat file

Hi,

resolved this issue by putting the 'Commit every' value to 1 in output component used.

And it is working fine now.

Employee

Re: Loading only correct data from flat file to azure SQL Data warehouse and storing rejected data into the flat file

Perfect!

 

Both approaches can resolve the issue. Could you please mark the posts which are having solutions to your original query so that it will help other Talend Community members during their reference?

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now