topic Re: AWS S3 file check in Talend Studio

AWS S3 file check

rishind — Fri, 15 Nov 2024 21:16:37 GMT

Hi ,

I am reading files from AWS S3 bucket.

There is no fix arrival timing for the files .

So as a file arrive ,I want to trigger the job.

How can I achieve this in Talend? Is there any component like tfileexist?

Re: AWS S3 file check

Anonymous — Mon, 06 Nov 2023 09:32:59 GMT

Hello

There is no a tS3FileExist component, you can try to use tinfiniteLoop+tS3list to check if files exist at regular intervals, then process the file and delete it if needed.

Regards

Shong

Re: AWS S3 file check

rishind — Mon, 06 Nov 2023 10:07:55 GMT

@Shicong Hong thanks for replying

In my case ,i cannot schedule it at an interval

It has to trigger whenever a file arrive .(without delay)

Files can arrive in 1 minute or 5 minutes or 7 minutes.

so there is no fixed schedule.

Re: AWS S3 file check

Anonymous — Wed, 08 Nov 2023 08:33:09 GMT

I think you need to find a way to monitor the S3 file, see the discussions in this page. Obviously Talend is not such a tool. What talend components can do is upload, download or delete S3 files.