topic Wait for File on a hdfs directory in Talend Studio

Wait for File on a hdfs directory

Teguard_Dev — Sat, 16 Nov 2024 04:42:29 GMT

Hello,

I'm using Talend Studio Big Data in the licensed version and am currently trying to switch over from a filesystem based ETL Process to HDFS based.

As of now, I'm using Wait for File to watch a File Directory for new Files to trigger my ETL Process / trigger a Input Row, as seen below.

Is there a way to replicate this using hdfs?

I'd be grateful for any Insights and help.

Cheers

manodwhb — Thu, 12 Sep 2019 07:31:28 GMT

@KS_UKHD , you can use tHDFSExists,check the below link to know more about this component.

Teguard_Dev — Thu, 12 Sep 2019 08:12:22 GMT

Thanks for the response,

but isn't that Component only for checking a single File?

Could you give an example how to use this to overwatch a directory and react to Files being created?

Cheers