Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
My scenario is that every day source files will come to different dynamic date folders in S3. I need to pickup the files after last processed timestamp and get those files to use in the main flow job. I am using ts3list component to list the files with the prefix(I can't give complete path as the folder are with dynamic dates). After that in ts3get component, i have to get only files which are newer than the last processed timestamp. But not much options are available. I am able to provide only ts3list current key in the key section. With this i am getting older processed files as well. Also once i get proper file, i don't want to store in the local and process directly in the job. Please help me to achieve this scenario. Thanks!!
Okay. What about picking up files with particular prefix from dynamic dates folders in S3 instead?
With below flow, I am able to list the files and select particular files based upon filter component. But, it's just for listing.. I am not able to process the file in the job.
tS3List -> tIterateToFlow -> tFilterRow -> tLogRow
I am not able to link to tS3Get from tFilterRow. If i link using onComponentOk from tFilterRow, then it's just picking the last file instead of all the required files. If i link tS3Get from tS3List, i am getting all the files instead of required files.
Please help.
@gopal16, after tlogrow use the tjavarow and set the filename as context and use ts3get component from tjavarow using on component ok and you will the required files.
Not able to link from tJavaRow to tS3Get with Main connection. Only onComponentOk is allowed and with that able to get only one last file. Not all the required files.
@gopal16 , yes you need to use on component ok to connect. to tlogrow,how many files it was getting?
To tLogRow i am gettting 10+ records with filenames
@gopal16 ,if that is the you will get all the files with the desgin of what i told. please verify.