Six Stars

Need info about tBigQueryOutput error

Hi,

I wish to load a table into BigQuery and am using tbigqueryoutput component to do the same. I thought, all that required was the dataset and the table name where it should be loaded and the data that is coming from talend.

But I see there are 2 other different parameters that are requested for

1. Local File Name

2. File on Google Storage Configuration.

My queries are

1. Why would there must be a local file name storing locally the output file to be pushed into bigquery? What do I have to do, if I do not wish to store any such details locally at all.

 bigquery_config.JPG

 

2. What content does the Google storage configuration exactly contain? Where is this file created?

Because with out this file the loading is not happening. Where in Big Query do I have to put the details of the google storage and what kind of details should exactly be containing into it.

 

Lastly,

I tried pushing records from talend into a Bigquery, but it keeps asking for the bucket details. Can anyone of you explain what kind of details should be present in the gs://<<bucket_Name>>/<filename_to_be_used_in_talend>>. PFB the screenshot of what parameters do i really mean.

13 REPLIES
Four Stars

Re: Need info about tBigQueryOutput error

Hi tsreenath,

 

this components put your local file in a bucket in google cloud storage and then import into bigquery table.

 

API Gloud storage and bucket must be activated and created.

 

It's clear ?

Six Stars

Re: Need info about tBigQueryOutput error

Do you mean Cloud Storage JSON API?

Also do we require any other APIs such as BigQuery API, Cloud Storage Service, Cloud Data Store API for loading the data from Talend into Talend BigQuery?


Four Stars

Re: Need info about tBigQueryOutput error

You need BigQuery & Google Cloud Storage

 

Image 696.png

 

And for acccess, we need Id client + access key

 

Image 697.png

 

For Cloud Storage, go to GCP Console and generate key

 

https://console.cloud.google.com/storage/settings?project=YOUR_PROJECT

 

Image 699.png

 

 

Six Stars

Re: Need info about tBigQueryOutput error

I already have the following details
Connection Details
1. Client ID
2. Client Secret
3. Project ID
4. Authorization Code

Google Storage Configuration
1. Access Key
2. Secret Key
3. Bucket


And I can able to extract the data from Google BigQuery (using tbigqueryinput component). Only problem I face is while loading the data into Big Query (i.e. using tbigqueryoutput component).



I do not know what exactly the Google storage file mean and what are required from Google cloud side (any api/access) to load the data into bigquery from Talend successfully. Can any of you confirm me any api or service required to get enable for loading bigquery through Talend.




Four Stars

Re: Need info about tBigQueryOutput error

Google Storage is used to copy file in Google before to load in Bigquery. The job load file from Google Cloud storage and not your file in local.

We need only Google Cloud Storage api.

Do you test to run your job, do you have error ?
Six Stars

Re: Need info about tBigQueryOutput error

Hi,

PFB the actual error which I am getting, can you please tell me what exactly is needed to be done to get this error resolved. With this message, Can I, in anyways assume the Google Cloud Storage API is already available or not?



[cid:image001.png@01D2DA34.5EA8EFD0]


Four Stars

Re: Need info about tBigQueryOutput error

Can you re-upload your image ?
Six Stars

Re: Need info about tBigQueryOutput error

 
Four Stars

Re: Need info about tBigQueryOutput error

can you add a print screen for your bucket ?
Six Stars

Re: Need info about tBigQueryOutput error

 
Four Stars

Re: Need info about tBigQueryOutput error

why file has not the same file
In your bucket CurriculumTemplate.json, in the error talend curriculum_template.json
Six Stars

Re: Need info about tBigQueryOutput error

Hi, it is a file created by me. May I know what will be the content of the file usually. Also, is it possible to load the data directly into Big Query as it seems the data is not stored in Cloud Storage at all. May I know what is to be done in that case?

Two Stars

Re: Need info about tBigQueryOutput error

totally clear

thay man hinh iphone 6