topic Re: Moving data around on on Azure Blob in Talend Studio

Moving data around on on Azure Blob

gurn — Fri, 15 Nov 2024 22:24:08 GMT

I'm after some advice on how best to move data around on Azure Blob. I'm new to Azure and so need to make sure we put in the right components for this. I'm looking to shift files from a landing folder into an archive folder.

Thoughts would be great...

Thanks

Re: Moving data around on on Azure Blob

InfoCraft — Wed, 02 Nov 2022 14:01:28 GMT

Hi @Dave Gurney,

you can use tAzureStoragePut.

Or check this https://community.talend.com/s/feed/0D55b00007Ro6n3CAB?language=en_US

Kind regards,

😷19.

Re: Moving data around on on Azure Blob

gurn — Wed, 02 Nov 2022 23:14:32 GMT

Thanks @not specified not specified

Unfortunately, I've hit another snag with an error thrown on a tAzureStorageList and so any pointers would be really useful...

---

Exception in component tAzureStorageList_1 (azure_move_test)

java.util.NoSuchElementException: An error occurred while enumerating the result, check the original exception for details.

at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:113)

at org.talend.components.azurestorage.blob.runtime.AzureStorageListReader.start(AzureStorageListReader.java:79)

at org.talend.codegen.flowvariables.runtime.FlowVariablesReader.start(FlowVariablesReader.java:73)

at evelyn_dev.azure_move_test_0_1.azure_move_test.tAzureStorageList_1Process(azure_move_test.java:704)

at evelyn_dev.azure_move_test_0_1.azure_move_test.runJobInTOS(azure_move_test.java:2367)

at evelyn_dev.azure_move_test_0_1.azure_move_test.main(azure_move_test.java:2155)

Caused by: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.

at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)

at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)

at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:196)

at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySegmentedIterator.java:109)

Re: Moving data around on on Azure Blob

Anonymous — Thu, 03 Nov 2022 05:32:35 GMT

Hello,

Please check and make sure that the folder path, file names and Azure storage signature are correct.

Could you please post your job design screenshots here so that we could get more information from it?

Best regards

Sabrina

Re: Moving data around on on Azure Blob

gurn — Mon, 07 Nov 2022 22:53:03 GMT

@Xiaodi Shi here you go - the folder path is correct and my connection is working - I'm using a connection with the Tenant ID, Client ID and Client Secret set.

I've extended Blobfilter to include the actual filename I've trying to list...

Re: Moving data around on on Azure Blob

InfoCraft — Tue, 08 Nov 2022 18:15:08 GMT

Hi @Dave Gurney,

that means, the container name do not exists.

Checks the correct container name. and if you are using Microsoft Azure Storage Explorer, take a screenshot of the container tree

Kind regards,

😷19.

Re: Moving data around on on Azure Blob

gurn — Mon, 21 Nov 2022 13:55:33 GMT

Thanks for all the feedback. After digging around it was found that the reason for the issue was related to not having a .* at the beginning of the pattern string to force the recursive search down the directory tree for the file.

Dave

Re: Moving data around on on Azure Blob

Anonymous — Tue, 22 Nov 2022 07:52:24 GMT

Hello,

Great you found the root cause and sharing your solution with us on Community.

Best regards

Sabrina