topic Re: Need info about tBigQueryOutput error in Talend Studio

Need info about tBigQueryOutput error

Anonymous — Sat, 16 Nov 2024 09:42:48 GMT

Hi,

I wish to load a table into BigQuery and am using tbigqueryoutput component to do the same. I thought, all that required was the dataset and the table name where it should be loaded and the data that is coming from talend.

But I see there are 2 other different parameters that are requested for

1. Local File Name

2. File on Google Storage Configuration.

My queries are

1. Why would there must be a local file name storing locally the output file to be pushed into bigquery? What do I have to do, if I do not wish to store any such details locally at all.

2. What content does the Google storage configuration exactly contain? Where is this file created?

Because with out this file the loading is not happening. Where in Big Query do I have to put the details of the google storage and what kind of details should exactly be containing into it.

Lastly,

I tried pushing records from talend into a Bigquery, but it keeps asking for the bucket details. Can anyone of you explain what kind of details should be present in the gs://<<bucket_Name>>/<filename_to_be_used_in_talend>>. PFB the screenshot of what parameters do i really mean.

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 10:53:18 GMT

Hi tsreenath,

this components put your local file in a bucket in google cloud storage and then import into bigquery table.

API Gloud storage and bucket must be activated and created.

It's clear ?

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 11:22:30 GMT

Do you mean Cloud Storage JSON API?

Also do we require any other APIs such as BigQuery API, Cloud Storage Service, Cloud Data Store API for loading the data from Talend into Talend BigQuery?

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 11:29:03 GMT

You need BigQuery & Google Cloud Storage

And for acccess, we need Id client + access key

For Cloud Storage, go to GCP Console and generate key

https://console.cloud.google.com/storage/settings?project=YOUR_PROJECT

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 11:46:30 GMT

I already have the following details
Connection Details
1. Client ID
2. Client Secret
3. Project ID
4. Authorization Code

Google Storage Configuration
1. Access Key
2. Secret Key
3. Bucket

And I can able to extract the data from Google BigQuery (using tbigqueryinput component). Only problem I face is while loading the data into Big Query (i.e. using tbigqueryoutput component).

I do not know what exactly the Google storage file mean and what are required from Google cloud side (any api/access) to load the data into bigquery from Talend successfully. Can any of you confirm me any api or service required to get enable for loading bigquery through Talend.

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 11:54:13 GMT

Google Storage is used to copy file in Google before to load in Bigquery. The job load file from Google Cloud storage and not your file in local.

We need only Google Cloud Storage api.

Do you test to run your job, do you have error ?

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 13:08:30 GMT

Hi,

PFB the actual error which I am getting, can you please tell me what exactly is needed to be done to get this error resolved. With this message, Can I, in anyways assume the Google Cloud Storage API is already available or not?

[cid:image001.png@01D2DA34.5EA8EFD0]

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 13:35:45 GMT

Can you re-upload your image ?

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 14:25:12 GMT

can you add a print screen for your bucket ?

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 15:21:30 GMT

why file has not the same file
In your bucket CurriculumTemplate.json, in the error talend curriculum_template.json

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 31 May 2017 18:07:31 GMT

Hi, it is a file created by me. May I know what will be the content of the file usually. Also, is it possible to load the data directly into Big Query as it seems the data is not stored in Cloud Storage at all. May I know what is to be done in that case?

Re: Need info about tBigQueryOutput error

Anonymous — Thu, 01 Jun 2017 04:58:15 GMT

totally clear

thay man hinh iphone 6

Re: Need info about tBigQueryOutput error

Anonymous — Wed, 20 Mar 2019 16:52:54 GMT

About storing file locally in LocalFilename is there a way to avoid this as I am using talend cloud instance to run jobs and it keeps telling that the path doesn't eist.