I need to extract data from big query for last 2 years.
I populated a table with dates with last 2 years.
Now I want to read date from this table and get daily data from big query for last 2 years.
What would be the best way iterate this job?
PS. Table_date_range in big query failed because of too much data.
If I understand your request well, you need to iterate each date and pass it to the query as a parameter to query only the data based on the current date.
Take a look at the tFlowToIterate component documentation and learn this component. eg:
Read date from data source--main(row1)--tFlowToIterate--iterate-->tBigQueryInput....
on tBigQueryInput, you can access the current value of date_column with this expression:
Introduction to Talend Open Studio for Data Integration.
Practical steps to developing your data integration strategy.
Create systems and workflow to manage clean data ingestion and data transformation.