Streaming Data from Azure Blob Storage using Cloud Engine

nickheidke — Sat, 16 Nov 2024 04:11:40 GMT

My use case is a need to take an Excel or AVRO file from Azure Blob Storage and load it into a SQL Database. In the past, when working with a Talend Remote Engine, I would use a tAzureStorageGet component to download the file locally. Then a separate component to open the local file and stream it into my destination. However, I have a new requirement to use a Talend Cloud Engine rather than a Talend Remote Engine. As such, I don't believe I have access to any local file system (please correct me if I'm wrong). Is there a component or technique to stream the data from an Azure Blob Storage object without first downloading it locally?

Re: Streaming Data from Azure Blob Storage using Cloud Engine

Anonymous — Fri, 08 Nov 2019 03:49:36 GMT

Hi
You can still access local file system when using Talend Cloud Engine, such as '/tmp/', and you can delete the file after it is processed in the end of job. Can you try and let me know if it works?

Regards
Shong

topic Streaming Data from Azure Blob Storage using Cloud Engine in Talend Studio

Streaming Data from Azure Blob Storage using Cloud Engine

Re: Streaming Data from Azure Blob Storage using Cloud Engine