Bigquery Table Handling Error

[Anonymous]

Hi,
I am trying to connect to Google Big query table from Talend. While fetching the data from Big query I am getting the below error.
Exception in component tBigQueryInput_3
com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
{
"code" : 403,
"errors" : ,
"message" : "Response too large to return."
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:159)
at com.google.api.client.googleapis.json.GoogleJsonResponseException.execute(GoogleJsonResponseException.java:187)
at com.google.api.client.googleapis.services.GoogleClient.executeUnparsed(GoogleClient.java:115)
at com.google.api.client.http.json.JsonHttpRequest.executeUnparsed(JsonHttpRequest.java:112)
at com.google.api.services.bigquery.Bigquery$Jobs$GetQueryResults.execute(Bigquery.java:1755)
at tutorials.mqs_tgt_0_1.MQS_TGT.tBigQueryInput_3Process(MQS_TGT.java:2256)
at tutorials.mqs_tgt_0_1.MQS_TGT.runJobInTOS(MQS_TGT.java:6135)
at tutorials.mqs_tgt_0_1.MQS_TGT.main(MQS_TGT.java:5960)
Is there way to overcome the above issue.
While querying the data directly in Google Biquery we have option to allow larger results. But there is no option available in Talend tBigQueryInput component to allow larger result from Bigquery table.
Thanks
Muthuraj

[Anonymous]

Hi,
Could you indicate the build version you're using? Have you checked the jira issue https://jira.talendforge.org/browse/TDI-28445?
Best regards
Sabrina

[Anonymous]

Hi,
Please find the below details.
Version: 5.3.1
Build id: r104014
Regards
Muthuraj

[Anonymous]

Hi,
Could you please also show us your tBigQueryInput query?
Best regards
Sabrina

[Anonymous]

Hi Sabrina,
I was in vacation for a month. Please find the query i used for the job.
"select * from as TR1 JOIN each as TH4 on TR1.TRACK_ID = TH4.TRACK_ID where DATE(TR1.INSERT_DATETIME) = DATE(DATE_ADD(TIMESTAMP(CURRENT_DATE()), -1, 'DAY')) ".
The source tables have around 100 million rows per day. So I need to extract the data and load into another table with some additional calculation. But am not able to fetch the complete data from Bigquery table using talend. It is throwing error.
Please let me know if you find anything.
Thanks & Regards
Muthuraj

[_AnonymousUser]

I am getting the following JSON exception while I am executing the query using the java application for Big-Query API. Following is the exception I am getting:

    {
    "error": {
    "errors": ,
"code": 403,
"message": "Access Denied: Job eco-span-847:job_LyHmZIvlY1_0J8JQ3pxThEBf19I: The user does not have permission to run a query in project eco-span-847"
}
}

[Anonymous]

Where can we set "allow-large-results" BQ parameter in tBigQueryInput component? Or how can we export a big table to csv (in GCS) without this issue?

[Anonymous]

Hi,
Have you already tried it on latest released version 5.6.1 to see if this issue repro?
This issue "tBigQueryInput does not return all rows if result is too big (BigQuery input component)" has been fixed in 5.2.4, 5.3.2, 5.4.2, 5.6.0M0a.
https://jira.talendforge.org/browse/TDI-28445

Best regards
Sabrina

[Anonymous]

HI All,
Could you please help me how did you resolve this issue. I am using version 5.6.2 but still I am facing the same issue. Where can I set the property Allow_large_results.
Thanks
Manoj