How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration

Highlighted
Four Stars

How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration

Hi,

We are using Talend cloud Studio to develop the jobs, then we are required to publish them in TIC. As we are using the Google Big Query  as our DB (Source & Destination), We are using the tBigqueryInput & tBigqueryOutput component. Both components require local files, as the input component need the token file and output component need local file to keep the data file , before uploading the dataset to cloud storage or insert them into google big query table. 

 

As mentioned earlier, we need to publish them to Talend Cloud and carry out the loads in TIC. Now we need to know, how to eliminate the local file use, is possible to use only the cloud storage, we have access to google cloud storage, where we can store the files.

Using Talend Cloud Studio 6.5  JDBC-ELT (not supporting big query datatype) and JDBC is (slow) (Simba jdbc), As we need to  process 500 million records in few tables and also we need a solution to eliminate local file .

As our requirement  is to have the ETL, only in Talend cloud platform and use the talend cloud engine. (no remote engine). So need help, to resolve the issue

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Talend Cloud Developer Series – Defining Metadata

This video focuses on different methods of adding metadata to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Updating Context Variables

This video will show you how to add context parameters to a job in Talend Cloud

Watch Now

Talend Cloud Developer Series – Deploying First Job to Cloud

This video will show you how to run a job in Studio and then publish that job to Talend Cloud

Watch Now