topic Re: To trigger talend Job based on S3 bucket event notifications in Talend Studio

To trigger talend Job based on S3 bucket event notifications

Anonymous — Sat, 16 Nov 2024 09:29:35 GMT

Hi,

We have data coming into S3 ,And based on File creation , we want to track the s3 event notifications and trigger a talend job.For this we are pang the below design:

S3 bucket -> Event Notification --> SNS -- SQS Queue --> for each event trigger the Talend job which reads data from S3.

i am just stuck in the last part.how shd i read the each message and trigger the talend child job? Should i read the meesage create a file and keep a file watcher and then kick off.or is there anyway where i can loop the job through the each mesage..

require a Wise solution for this as the performance peeps in when i loop thru each message.

Re: To trigger talend Job based on S3 bucket event notifications

Anonymous — Mon, 24 Jul 2017 20:31:46 GMT

Follow this https://help.talend.com/reader/q456yoRkDysTdtIPChwkJA/tDGpKsjQFdn7~JgqDHP2iw

You do not need a file watcher. The SQS Message Notification should be enough if you provide the path to the S3 file in the message. The job can read the SQS Message and get the path of the file and then process it.

Re: To trigger talend Job based on S3 bucket event notifications

Anonymous — Fri, 28 Jul 2017 07:58:03 GMT

Thanks for the information.One technical Doubt i have is..SQS Queue will have the event notifications,How should i design my Job to have a conditional trigger?

I have one job developed which downlods from S3 ,but hw shd i design the job to integrae it with SQS ..initial step.

I wil configure my TSQSINput to read message ,but hw shd i loop my code based on one message(event),take the filename from event and run S3get..

I am just not able to visualise this. and stuck here.can anyone help pls?