topic Re: tFileList and HDFS connection in Talend Studio

tFileList and HDFS connection

Anonymous — Sat, 16 Nov 2024 10:29:18 GMT

Hi,
is there a way to do a tFileList utilizing an existing connection to a hadoop cluster? Would like to return a list and then decide whether to open file or not depending upon name.
thanks,
Bob

Re: tFileList and HDFS connection

Anonymous — Wed, 27 Jul 2016 09:41:40 GMT

Hi,

is there a way to do a tFileList utilizing an existing connection to a hadoop cluster? Would like to return a list and then decide whether to open file or not depending upon name.

The component tfilelist can not retrieve a set of files or folders on hadoop cluster server directly.
Here is a TalendHelpCenter:tHDFSList component which can retrieve a list of files or folders based on a filemask pattern and iterates on each unity.
Best regards
Sabrina

Re: tFileList and HDFS connection

Anonymous — Thu, 02 Mar 2017 09:10:04 GMT

Hi,
Is there a tHDFSList equivalent for Spark?
Thank you

Re: tFileList and HDFS connection

Anonymous — Thu, 02 Mar 2017 21:28:27 GMT

You would have to handle that in a Standard DI job then call spark big data job with tRunJob passing any context to the spark job as needed.