How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration

Highlighted
Four Stars

How to Eliminate Local dataset (file) in Big Query Output,bulkexec component while using the Talend Cloud Integration

Hi,

We are using Talend cloud Studio to develop the jobs, then we are required to publish them in TIC. As we are using the Google Big Query  as our DB (Source & Destination), We are using the tBigqueryInput & tBigqueryOutput component. Both components require local files, as the input component need the token file and output component need local file to keep the data file , before uploading the dataset to cloud storage or insert them into google big query table. 

 

As mentioned earlier, we need to publish them to Talend Cloud and carry out the loads in TIC. Now we need to know, how to eliminate the local file use, is possible to use only the cloud storage, we have access to google cloud storage, where we can store the files.

Using Talend Cloud Studio 6.5  JDBC-ELT (not supporting big query datatype) and JDBC is (slow) (Simba jdbc), As we need to  process 500 million records in few tables and also we need a solution to eliminate local file .

As our requirement  is to have the ETL, only in Talend cloud platform and use the talend cloud engine. (no remote engine). So need help, to resolve the issue

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Self-service Talend Migration: Moving from On-Premises to the Cloud

Move from On-Premises to the Cloud by following the advice of experts

Read Now

Modern Data Engineering in the Cloud

Learn about modern data engineering in the Cloud

Watch Now

How to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud reg...

Learn how to deploy Talend Jobs as Docker images to Amazon, Azure and Google Cloud registries

Blog