topic Google Cloud Storage File Streaming in Data Quality

Google Cloud Storage File Streaming

Anonymous — Fri, 27 Apr 2018 16:35:36 GMT

Hello,

I have a bunch of files in Google Cloud Storage that I'd like to stream into my job. Is it possible to stream in the byte array data for each file?

Currently I'm using tGSGet to download the files to my local machine first and then looping through them to get byte array data. Ideally, I'd be able to loop through the files without downloading them first.

Re: Google Cloud Storage File Streaming

Anonymous — Thu, 03 May 2018 10:13:40 GMT

Hello,

Currently, you have to download the files to your local machine when use tGSGet component. It retrieves objects which match the specified criteria from Google Cloud Storage and outputs them to a local directory.

A new feature of Google Cloud Storage in Apache Spark Batch/Streaming is available in V 6.4.

Please have a look at this online document about:TalendHelpCenter:tGSConfiguration properties for Apache Spark Streaming

Best regards

Sabrina

Re: Google Cloud Storage File Streaming

Anonymous — Thu, 03 May 2018 16:22:13 GMT

I am using a BigQuery query to get a list of bucket/key values. I want to iterate on those to pull files.

tBigQueryInput --> tflowtoiterate -->tgsget

If I hard code Bucket name: "mybucket" and Keyprefix: "myfolder/mydocument" it works as expected and downloads the file to my local machine.

However, toflowtoiterate saves my bucket and key to global variables first. When I use those global variables in tGSget it gives me a null error.

How can I get this to work?

1.png
2.png