Bigquery Table Handling Error

Five Stars

Bigquery Table Handling Error

Hi,
I am trying to connect to Google Big query table from Talend. While fetching the data from Big query I am getting the below error.
Exception in component tBigQueryInput_3
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
"code" : 403,
"errors" : ,
"message" : "Response too large to return."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:159)
at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:187)
at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:115)
at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:112)
at com.google.api.services.bigquery.Bigquery$Jobs$GetQueryResults.execute(Bigquery.java:1755)
at tutorials.mqs_tgt_0_1.MQS_TGT.tBigQueryInput_3Process(MQS_TGT.java:2256)
at tutorials.mqs_tgt_0_1.MQS_TGT.runJobInTOS(MQS_TGT.java:6135)
at tutorials.mqs_tgt_0_1.MQS_TGT.main(MQS_TGT.java:5960)
Is there way to overcome the above issue.
While querying the data directly in Google Biquery we have option to allow larger results. But there is no option available in Talend tBigQueryInput component to allow larger result from Bigquery table.
Thanks
Muthuraj
Moderator

Re: Bigquery Table Handling Error

Hi,
Could you indicate the build version you're using? Have you checked the jira issue https://jira.talendforge.org/browse/TDI-28445?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Bigquery Table Handling Error

Hi,
Please find the below details.
Version: 5.3.1
Build id: r104014
Regards
Muthuraj
Moderator

Re: Bigquery Table Handling Error

Hi,
Could you please also show us your tBigQueryInput query?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Bigquery Table Handling Error

Hi Sabrina,
I was in vacation for a month. Please find the query i used for the job.
"select * from as TR1 JOIN each as TH4 on TR1.TRACK_ID = TH4.TRACK_ID where DATE(TR1.INSERT_DATETIME) = DATE(DATE_ADD(TIMESTAMP(CURRENT_DATE()), -1, 'DAY')) ".
The source tables have around 100 million rows per day. So I need to extract the data and load into another table with some additional calculation. But am not able to fetch the complete data from Bigquery table using talend. It is throwing error.
Please let me know if you find anything.
Thanks & Regards
Muthuraj
One Star

Re: Bigquery Table Handling Error

I am getting the following JSON exception while I am executing the query using the java application for Big-Query API. Following is the exception I am getting:
    {
   "error": {
   "errors": ,
"code": 403,
"message": "Access Denied: Job eco-span-847:job_LyHmZIvlY1_0J8JQ3pxThEBf19I: The user does not have permission to run a query in project eco-span-847"
}
}
One Star

Re: Bigquery Table Handling Error

Where can we set "allow-large-results" BQ parameter in tBigQueryInput component? Or how can we export a big table to csv (in GCS) without this issue?
Moderator

Re: Bigquery Table Handling Error

Hi,
Have you already tried it on latest released version 5.6.1 to see if this issue repro?
This issue "tBigQueryInput does not return all rows if result is too big (BigQuery input component)" has been fixed in 5.2.4, 5.3.2, 5.4.2, 5.6.0M0a.
https://jira.talendforge.org/browse/TDI-28445

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Bigquery Table Handling Error

HI All,
Could you please help me how did you resolve this issue. I am using version 5.6.2 but still I am facing the same issue. Where can I set the property Allow_large_results.
Thanks
Manoj

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog