database input NB_LINE variable is always 'int'. Does it work when the table has 3 billion rows?

Four Stars

database input NB_LINE variable is always 'int'. Does it work when the table has 3 billion rows?

I am designing a job that exports rows from a informix table.

Lets say i am selecting from an informix table with 3 billion rows. The  variable tInformixInput_1_NB_LINE is declared as int type.

The component will fail when total number of rows selected gets beyond the integer range. What are the design considerations for this scenario?

Forteen Stars

Re: database input NB_LINE variable is always 'int'. Does it work when the table has 3 billion rows?

if it just theoretic question, the answer - do not know, need test.

 

but if it practical question - if any ETL process, excluding initial loading, have regular input for billions of rows   ... it look like something wrong in process design. 

 

In any case You always can count records by other methods.

-----------
Four Stars

Re: database input NB_LINE variable is always 'int'. Does it work when the table has 3 billion rows?

Thanks for your reply and I completely agree with you.

Theoretically, what are my options if I need to initially load more than 3 billion rows from DB2/zos database using talend?

I can only think of doing it in batches, i.e split the primary key range to manageable chunks and load the data.
Forteen Stars

Re: database input NB_LINE variable is always 'int'. Does it work when the table has 3 billion rows?

by the many reasons - the best choice for huge database will be avoiding direct loading whole table in 1 step

 

Technics could be different, all depends from real tasks and could be adjusted

-----------

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog