topic Re: Talend 8 Job reading all messages in Azure Storage Queue in Talend Studio

Talend 8 Job reading all messages in Azure Storage Queue

GeoNL — Mon, 29 Apr 2024 20:37:53 GMT

In a Talend 8 job I need to read messages form an Azure Storage Queue.

The options for this are tAzureStorageQueueInput and tAzureStorageQueueInputLoop

The problem with the tAzureStorageQueueInput is that it can retrieve a maximum of 32 messages per run instead of just collecting all that is there in the queue.

The problem with the tAzureStorageQueueInputLoop is that it never breaks out of the loop, how would I know it can safely terminate? In the documentation there is nothing mentioned on how to stop this infinite loop.

Documentation used: https://help.talend.com/r/en-US/8.0/azure-storage-queue/tazurestoragequeueinputloop

Re: Talend 8 Job reading all messages in Azure Storage Queue

GeoNL — Wed, 01 May 2024 06:40:22 GMT

Workaround used:

PreJob define a globalVar boolean 'done' as false,
Start with a tLoop while 'done' is false and iterate over tAzureStorageQueueInput + rest of job,
tAzureStorageQueueInput get's an onComponentOk where you check the NB_LINE
- If that is 0 you set the variable 'done' to true
- theoretically if it is less than the __numberOfMessages__ you could already set 'done' true

Re: Talend 8 Job reading all messages in Azure Storage Queue

Xiaodi_Shi — Tue, 07 May 2024 03:43:17 GMT

Hello,

Thanks for sharing your suggestion and good solution with us on Community. We will send it to DOCT team to update the documentation if necessary.

Best regards

Sabrina