Streaming over sql table/query results row by row. Is it possible?

One Star

Streaming over sql table/query results row by row. Is it possible?

Greeetings,
I have a number of large tables in an Microsoft Sql Server database. I would like to use Talend ETL to build an ETL pipeline, but I would like to create a job so that the tables from the database are read row by row, very likely using a db cursor.
Each row should then go through various transformations.

I will certainly hit memory limits if I try to pull all source data from the source database and this would also effect the performance of the source database for other users.

I could not find any documents/comments that clearly explains if this is possible with Microsoft Sql server. Which component would allow me do this, if there is one? How should the component be configured? Some pointers would help a lot.

Regards
Seref
Seventeen Stars

Re: Streaming over sql table/query results row by row. Is it possible?

It is of course possible, actually this is what the Talend database input component supposed to do. Memory issues usually occurs while reading a large number of wide lookup records. Try the additional JDBC property: responseBuffering=adaptive
Microsoft explained it here:
http://msdn.microsoft.com/en-us/library/bb879937.aspx 
One Star

Re: Streaming over sql table/query results row by row. Is it possible?

Thanks,
According to Microsoft documentation, adaptive response buffering is the default behaviour for jdbc drivers > 1.2
I think another option for me is to look at the source of the input component and see what it is doing.

All the best
Seref

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Have you checked out Talend’s 2019 Summer release yet?

Find out about Talend's 2019 Summer release

Blog

Talend Summer 2019 – What’s New?

Talend continues to revolutionize how businesses leverage speed and manage scale

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog