Talend Connect
Virtual Summit
JOIN US!
And visit the Customer
& Community Lounge.
May 27-28, wherever you are.

Alternatives to BigQuery database in tDBInput and tDBOutput components

Highlighted
Four Stars

Alternatives to BigQuery database in tDBInput and tDBOutput components

Hello all,

 

I have a Talend job that is responsible for reading data from a SELECT query and create a table to store the data from the query output on various types of database such as SQL Server, Redshift, BigQuery. The SELECT query is varied for each run and is stored as a global variable passed from a central Meta DB; it gets data from different sources and can result in various schema. The job includes 2 components, tDBInput with a Dynamic Column schema, and tDBOutput with drop/re-create table as needed from the Dynamic Column schema.

 

I can make it works for most target databases but BigQuery because both tDBInput and tDBOuput does not have BigQuery in the Database drop-out.

 

I've explored other workarounds:

  1. Use JDBC driver (Simba) in tDBInput & tDBOutput, but it doesn't support the options of drop-recreate table as needed in the output component, and looks like the Dynamic Field in tDBInput doesn't work as well
  2. Try tBigQueryInput component in place of tDBInput, buttBigQueryInput component does not support Dynamic field in schema

 

Has anyone experienced the same issue with BigQuery? Is there any workaround to add BigQuery into tDBInput and tDBOutput components?

 

I'm using Talend 7.1 Enterprise version, Simba driver 4.2

 

Thanks,

Tam

 

Highlighted
Moderator

Re: Alternatives to BigQuery database in tDBInput and tDBOutput components

Hello,

tBigQueryXXX components don't support for dynamic schema.

Could you load multiple files with different schema structure into tFileOutputDelimited first then migrating data from one .csv file(tFileInputFullRow) to tBigQueryOutput component?
Best regards
Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Highlighted
Four Stars

Re: Alternatives to BigQuery database in tDBInput and tDBOutput components

Hi,

 

Used JDBC driver (Simba) in tDBInput & tDBOutput instead of tBigQueryInput & tBigQueryOutput to connect Bigquery , are these components support DML actions like delete , insert and update. I am trying a job to run multiple dml statements run sequentially. 

 

As we don't see tDBRow in tBigQuery components, I am unable to run DML statements. 

My Target is execute DELETE STATMENT,INSERT STATEMENT & UPDATE statement in a single talend job.

 

Please help me.

 

Thanks

Chandra

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog