problem Talend with azure

[Anonymous]

Hi all,

I installed Talend Open Studio for Big Data to process my data with azure.

To start, I have to make a poc which allows the connection between talend and azure, then get the csv files that I have in HD Insight in azure, aggregate them with talend(tmap component) and put them in azure to finish.

For the moment I manage to recover the files from azure but then it is not possible to process them without storing them on my pc.

Do you have any idea how to do this ?

Thanks 😊 (and sorry for my english, i'm french)

[vapukov]

Hi,

unfortunately for Oen Studio - no way to process files without storing them somewhere

I'm not familiar with HD Insight, but as I can see from the documentation it provides SQL (Hive) access to the data

something like - https://docs.microsoft.com/en-us/azure/machine-learning/team-data-science-process/move-hive-tables 

but generally, based on the idea - Big Data usually really Big 🙂 Talend Open Studio (and any external tool) not the best tool for external data manipulation

it is good to load data into the big warehouse or data lake with some pre-processing