topic Re: Write files iteratively to Azure Blob Storage in Talend Studio

Write files iteratively to Azure Blob Storage

ANK1656078988 — Fri, 15 Nov 2024 22:47:37 GMT

Hi,

I am looking to randomly generate some data using the tRowGenerator component, write this into a file and upload this data to Azure Blob Storage container, at a 10 second interval

What is the best way to do this?

I am currently stuck at the point where I cant connect the "main" line of tfileoutoutdelimited and write into Azure.

The tAzureStoragePut component requires the all the files be stored in a directory before it can be uploaded to Azure. However I want the upload to happen iteratively, one file at a time.

Any suggestions?

Re: Write files iteratively to Azure Blob Storage

InfoCraft — Sat, 25 Jun 2022 12:50:09 GMT

Hi @Aravind NK,

*** tAzureStoragePut can send one or more file in the same time.

So, you have 2 solutions:

Send Files on by one(for each iteration)

Example: I'm able to send all files starting with test0 to the Azure Storage...

===> I don't understand why you iterate every 10 seconds!

Kind regards,

😷19.

Re: Write files iteratively to Azure Blob Storage

ANK1656078988 — Wed, 29 Jun 2022 06:24:24 GMT

Hi @not specified not specified Thank you for the response.

The reason behind iterating every 10 seconds is to simulate a streaming use case, where each file would represent a stream of data.

I am clear on the first option.

For the second one, will the tLoop component OnSubJobOk connector trigger tAzurePutStorage during every iteration?

Re: Write files iteratively to Azure Blob Storage

InfoCraft — Wed, 29 Jun 2022 09:19:47 GMT

Hi @Aravind NK,

OnComponentOK: will trigger the tAzureStoragePut after each itération.
OnSubjobOK triggers tAzureStoragePut at the end of all iterations

+++ gift_reminder,

OnSubjobOK and OnComponentOK are trigger links, which can link to another subjob. The main difference between them lies in the execution order of the linked subjob.

With OnSubjobOK, the linked subjob starts only when the previous subjob completely finishes.
With OnComponentOK, the linked subjob starts when the previous component finishes.

Kind regards,

😷19.