topic Re: How to process files only for the past hour? in Data Quality

How to process files only for the past hour?

Anonymous — Sat, 16 Nov 2024 09:26:04 GMT

I have continuous sensor data coming in every 5 mins in form of files. I want to pick files only for the past hour and do the required processing. for e.g: talend job runs at 12:01pm , it should pick all the files from 11:00 am to 12:00 pm only.
Can anyone please suggest the approach I should take to make this happen within talend.

Re: How to process files only for the past hour?

Anonymous — Tue, 08 Aug 2017 07:51:55 GMT

Hi,

Could you please elaborate your case with an example with input and expected output values?

Best regards

Sabrina

Re: How to process files only for the past hour?

vapukov — Tue, 08 Aug 2017 08:06:07 GMT

if file name include date time pattern, just use

tFileList -> iterate

in tFileList, use pattern like:

"commonpart-*"+TalendDate.formatDate("yyyyMMddHH",TalendDate.addDate(TalendDate.getCurrentDate(),-1,"HH"))+"*.csv"

if file names not include time pattern - need work with tSystem and bash, look there

https://stackoverflow.com/questions/26968159/list-of-files-modified-1-hour-before

than parse result

Re: How to process files only for the past hour?

Anonymous — Tue, 08 Aug 2017 14:05:44 GMT

I had initiated a post on https://stackoverflow.com/ .

https://stackoverflow.com/questions/45550213/how-to-process-files-only-for-the-past-hour-using-talend

I have reached to a point but facing issues(in comments)

Re: How to process files only for the past hour?

vapukov — Tue, 08 Aug 2017 21:42:33 GMT

no comments in thread (as I can see - You are move it to chat)

but in any case - suggested solution, no cover previous Hour,

If You check 1 Hr from current moment - it could have or duplicates, or holes. You never can run Job exactly every Hr

so You can mix recommendations and check not getCurrentDate() -1hr, but exactly low and high margins

more than 11:00:00 and less than 12:00:00

Re: How to process files only for the past hour?

Anonymous — Wed, 09 Aug 2017 18:40:30 GMT

Thanks Vapukov, currently i am going with getCurrentDate() -60mm.

Please suggest how can i specify 11:00:00 and less than 12:00:00. taking consideration that the job runs between the hourly intervals.

e.g: if the job runs at 11.20 then it should pick the lower limit as 11 and upper limit as 12