Iterate a job

Five Stars

Iterate a job

Hello,

I need to extract data from big query for last 2 years. 

I populated a table with dates with last 2 years. 

Now I want to read date from this table and get daily data from big query for last 2 years.

What would be the best way iterate this job?

PS. Table_date_range in big query failed because of too much data. 

 

Community Manager

Re: Iterate a job

Hi
If I understand your request well, you need to iterate each date and pass it to the query as a parameter to query only the data based on the current date.
Take a look at the tFlowToIterate component documentation and learn this component. eg:
Read date from data source--main(row1)--tFlowToIterate--iterate-->tBigQueryInput....

on tBigQueryInput, you can access the current value of date_column with this expression:
(java.util.Date)globalMap.get("row1.date_column")

Regards
Shong

----------------------------------------------------------
Talend | Data Agility for Modern Business

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.