We have the below scenario in our project. We have a S3 bucket. We recieve 3rd party files in that folder. we recieve hourly files in that folder. The number of files could also vary from 2 to 5 depending on the volume of the data.
The requirement is to extract these latest .csv files every hour and process them through Talend to redshift database. Can some one suggest how can we extract ONLY the latest files from S3 bucket out of all the files kept there? would appreciate any inputs for the same.
yes i agree. That is why i have the asked the client team to place the latest files in a new folder rather in same folder which has all the other files. Then the process can be easily automated.
