I am designing a job that exports rows from a informix table.
Lets say i am selecting from an informix table with 3 billion rows. The variable tInformixInput_1_NB_LINE is declared as int type.
The component will fail when total number of rows selected gets beyond the integer range. What are the design considerations for this scenario?
if it just theoretic question, the answer - do not know, need test.
but if it practical question - if any ETL process, excluding initial loading, have regular input for billions of rows ... it look like something wrong in process design.
In any case You always can count records by other methods.
by the many reasons - the best choice for huge database will be avoiding direct loading whole table in 1 step
Technics could be different, all depends from real tasks and could be adjusted
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Pick up some tips and tricks with Context Variables
Learn how media organizations have achieved success with Data Integration
Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend