Load data into DB if file exists in S3

[SR2210]

Hi,

Can anyone help to suggest how do I load data from S3 to PostgreSQL but only if the file is uploaded in S3 on today's date?

The file in S3 is not consistently uploaded every day. So currently I need to manually check if there is file today, then i run my talend job. If there's no file, then no need to run the job. 

I just want to know is there any component in Talend to check if the file exist in S3 then only the job will run, if there's no file then the job will not run.

Thank you in advance

[nivedhitha]

@SR2210 ,\

the easiest would be to use a tFileList after the tDropboxget, get the file names ((String)globalMap.get("tFileList_1_CURRENT_FILE")), check for the date and process the file like shown below

tJava1 does nothing. I have used it to just do an If condition. order:2 is the true case and the subjob executes if the condition is true. 

This job can be scheduled at fixed intervals and it would do nothing if the source file doesn't exist

[SR2210]

Hi @nivedhitha,

Thank you for your reply. I don't quite understand how to relate the tDropboxget and S3?

[nivedhitha]

My bad. Sorry.i meant s3get

[SR2210]

i see, at which component should i check the date?

[MattE]

There is a component called tS3List which if you supply it with a bucket name it will iterate through and list all the files in that bucket. tS3List has a variable called tS3List_1_CURRENT_KEY which is the filename. You could evaluate this variable and if it matches the filename you're looking for then carry on with the rest of the job.

[SR2210]

Hi @MattE,

My file name differs everyday. Example as below;

1. Nomination_220120.xlsx --(22 Jan 20)

2. Nomination_200120.xlsx --(20 Jan 20)

3. Nomination_261219.xlsx --(26 Dec 19)

"You could evaluate this variable and if it matches the filename you're looking for then carry on with the rest of the job."

This means that the part to check if it matches the filename is different for each day right? Because everyday I will look for different file name.This way I will not be able to schedule the job in TAC. Is there anyway to check the last modified date? For example if there's a file uploaded today then only the job will run. 

Thank you

[MattE · Accepted solution]

You can build the filename dynamically each day, this code

"Nomination_"+TalendDate.formatDate("ddMMyy", TalendDate.getCurrentDate())+".xlsx"

would build your filename in the specified format for the day it was run. 

You've then got some options. You could loop through the files in the bucket and only proceed if 

((String)globalMap.get("tS3List_1_CURRENT_KEY")) == "Nomination_"+TalendDate.formatDate("ddMMyy", TalendDate.getCurrentDate())+".xlsx"

or a more simple option is to put "Nomination_"+TalendDate.formatDate("ddMMyy", TalendDate.getCurrentDate())+".xlsx" in the Key of tS3Get so this component will only retrieve today's file and throw an error if it doesn't exist. 

[SR2210]

Thank you! it works