topic Retrieve the selected files from S3 bucket and process those in the job directly in Talend Studio

Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Sat, 16 Nov 2024 02:22:13 GMT

My scenario is that every day source files will come to different dynamic date folders in S3. I need to pickup the files after last processed timestamp and get those files to use in the main flow job. I am using ts3list component to list the files with the prefix(I can't give complete path as the folder are with dynamic dates). After that in ts3get component, i have to get only files which are newer than the last processed timestamp. But not much options are available. I am able to provide only ts3list current key in the key section. With this i am getting older processed files as well. Also once i get proper file, i don't want to store in the local and process directly in the job. Please help me to achieve this scenario. Thanks!!

Re: Retrieve the selected files from S3 bucket and process those in the job directly

Anonymous — Thu, 21 May 2020 10:25:15 GMT

Hi
I have seen the similar requirement open by other users, unfortunately, it is impossible to read the file directly on S3, you have to download the file to local system, then process it, and delete the file from local system after it is done if needed.

Regards
Shong

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Thu, 21 May 2020 10:27:07 GMT

Okay. What about picking up files with particular prefix from dynamic dates folders in S3 instead?

Re: Retrieve the selected files from S3 bucket and process those in the job directly

Anonymous — Thu, 21 May 2020 12:51:37 GMT

Hi
Set the key prefix to the parent folder which will contains the dynamic date folders, it will list all the sub-folders and files, then you can filter the files based on certain condition, such as file extension "*.txt".

Regards
Shong

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 03:19:27 GMT

With below flow, I am able to list the files and select particular files based upon filter component. But, it's just for listing.. I am not able to process the file in the job.

tS3List -> tIterateToFlow -> tFilterRow -> tLogRow

I am not able to link to tS3Get from tFilterRow. If i link using onComponentOk from tFilterRow, then it's just picking the last file instead of all the required files. If i link tS3Get from tS3List, i am getting all the files instead of required files.

Please help.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 05:17:05 GMT

@gopal16, after tlogrow use the tjavarow and set the filename as context and use ts3get component from tjavarow using on component ok and you will the required files.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 05:20:41 GMT

Not able to link from tJavaRow to tS3Get with Main connection. Only onComponentOk is allowed and with that able to get only one last file. Not all the required files.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 05:25:34 GMT

@gopal16 , yes you need to use on component ok to connect. to tlogrow,how many files it was getting?

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 05:26:46 GMT

To tLogRow i am gettting 10+ records with filenames

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 05:33:38 GMT

@gopal16 ,if that is the you will get all the files with the desgin of what i told. please verify.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 05:54:11 GMT

Sorry Manohar. With your approach, i am getting only one last file instead of 14 files. Below is the screenshot of that.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 05:58:20 GMT

I think i found the solution. Below job did work and able to filter and get all the required files.

Now, i need to work on loading these files directly to a table without placing in local using s3select option in tS3Get. If anyone succeeded on s3select option, please let me know.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 06:07:17 GMT

@gopal16 , you need to use tflowtoIterate pass the file by file to tS3Get.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 06:29:39 GMT

Yes, after using tflowtointerate, it did work as mentioned previously. Now, i need help to read the files that i got from tS3Get and load all into a table. All files are having same structure. So, can be loaded into single table.

When i tried the below, it's throwing error : Duplicate nested type row2Struct. And when i re-compiling, it's throwing other error saying one of the field cannot be resolved or is not a field. Actually i am not using that field in tFileInputDelimited. That's defined in tIterateToFlow component only. If i am not having tFileInputDelimited and tDBOutput components in this job, it's running fine. Not sure where is the problem.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 06:37:58 GMT

@gopal16 , til s3get no issue right,it is compilation issue you might have not configured correctly.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 06:55:13 GMT

till ts3get no issues. When i added tfileinputdelimited and tdboutput components, then it's throwing this error. In tfileinputdelimited, in filename place i am giving the same filename that mentioned while getting it from ts3get. And then simply connected from tfileinputdelimited to tdboutput for loading those files. I didn't change any other configuration.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 08:29:57 GMT

@gopal16 , you need to give the local filename along with in tfileinputdelimtted,do not use the global variabes of ts3list and check.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

gopal16 — Tue, 26 May 2020 08:38:18 GMT

Yes, I am using the local folder only in the filename. I am giving the same filename in tinputfiledelimited that's mentioned dynamically in File option of tS3Get as i can't hardcode the filename.

Re: Retrieve the selected files from S3 bucket and process those in the job directly

manodwhb — Tue, 26 May 2020 09:19:44 GMT

@gopal16 , can you share screenshot of tfileinputdelimitted

Re: Retrieve the selected files from S3 bucket and process those in the job directly

sushantk19 — Wed, 29 Sep 2021 13:37:59 GMT

@Manohar B : is there a way to get just the latest files from S3 if my client is placing all the files in same S3 bucket. File name changes every hour.