topic Re: Configure Talend to use ADLS in Talend Studio

Configure Talend to use ADLS

Anonymous — Sat, 16 Nov 2024 06:47:43 GMT

How do we correctly configure Talend to use ADLS for storage? The tHDFSConnection requires you to select a big data distro and version otherwise ADLS is not listed as a valid option. In this case ADLS is just a cloud based HDFS compliant file system and the big data distro option makes no sense.

Why do we have to select a distro and which one should we use for this component to work with both ADLS Gen1 and Gen2? Please tell me if I’m completely missing something here?

NB. I’m aware of the separate fact that Cloudera 5.11 introduced support for ADLS but this is only in relation to a Cloudera cluster using ADLS as its storage layer when using Hive, MapReduce, Impala, etc. This is obviously not related to using Talend to read / write to ADLS.

Re: Configure Talend to use ADLS

Anonymous — Wed, 30 Jan 2019 09:51:27 GMT

Hello,

Here exists a jira issue on talend bug tracker

https://jira.talendforge.org/browse/TBD-7116

Let us know if it is what you are looking for.

Best regards

Sabrina

Re: Configure Talend to use ADLS

Anonymous — Wed, 30 Jan 2019 12:11:41 GMT

Hi Sabrina,

No, this is not what I'm referring to. That ticket relates to finding
ADLS components on the palette. My issue relates to a lack of
instructions on how to correctly use those HDFS components to work with
ADLS due to their requirement to specify a specific Hadoop distribution.
Incidentally, that ticket remains as "unresolved" but I notice that when
I search for ADLS that the HDFS components are returned on the palette.
Hence you can probably have somebody test and confirm this and close
that ticket. Not that it helps my issue of course.
Best

Re: Configure Talend to use ADLS

Anonymous — Sat, 17 Aug 2019 21:21:17 GMT

You can pick Cloudera or Horton (aka HD Insight) obviously for you will be irrelevant, but the component will pick the right libraries for ADLS, in this case the best scenario is to have a unique component for ADLS and this component will come later on but in Parallel you can access to ADLS using this approach.

https://help.talend.com/reader/98wTdxOyJjhkll4BXt6Xcw/rzE1cW6_Jsslh~TM5GQ1Uw