topic Re: AWS s3 Filewatcher in Talend Studio

AWS s3 Filewatcher

Anonymous — Thu, 11 May 2017 07:03:30 GMT

Hi,

How to implement S3 filewatcher in talend..?

Thanks.

Re: AWS s3 Filewatcher

vapukov — Thu, 11 May 2017 11:10:40 GMT

do You want real watcher? ("real-time")

or just want from time to time check file exists or not?

Re: AWS s3 Filewatcher

vapukov — Thu, 11 May 2017 11:23:43 GMT

I suggest You check S3 notification system http://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html

I not test this way, but should be work - S3 -> SQS -> Talend

Re: AWS s3 Filewatcher

Anonymous — Thu, 11 May 2017 11:23:46 GMT

just want from time to time check file exists or not?

Re: AWS s3 Filewatcher

vapukov — Thu, 11 May 2017 11:27:16 GMT

tS3BucketExists - if need check Bucket

tS3List - uncheck "List all object" checkbox and test patterns for objects

Re: AWS s3 Filewatcher

Anonymous — Thu, 11 May 2017 14:04:52 GMT

Hi ,

I just want to check weather file is exist in s3 bukect or not around one hour means i thought to use the tsleep and tloop and iterate around one hour if file is not exist after one hour then we will populate the message file is not exist..

Re: AWS s3 Filewatcher

vapukov — Thu, 11 May 2017 22:34:47 GMT

You can use this components

Also - You can avoid tLoop and tSleep - just run You Job by scheduler. No reason to keep memory when it not necessary.

Other way (I use them some time) - install AWS cli on Talend Machine, and do this with command line interface with tSystem.

Example - I use this for fetch most resent backup from S3.

Re: AWS s3 Filewatcher

Anonymous — Fri, 12 May 2017 06:25:59 GMT

Other way (I use them some time) - install AWS cli on Talend Machine, and do this with command line interface with tSystem.

Example - I use this for fetch most resent backup from S3.

Hi ,

how to install AWS cli on Talend Machine..

THanks.

Re: AWS s3 Filewatcher

vapukov — Fri, 12 May 2017 07:43:09 GMT

they have excellent docs on site, all what you need - keys which will need install when setup client
for this reason better to create separate for Talend

http://docs.aws.amazon.com/cli/latest/userguide/installing.html