topic Re: How to iterate from ts3Get in Talend Studio

How to iterate from ts3Get

Anonymous — Sat, 16 Nov 2024 02:34:16 GMT

Scenario: I have a few files in S3 bucket, now I want to get only a single file at a time and process the file and then get another file again and process it again, this should continue for all the files present in the S3.

please find the job design.

Thanks in advance.

Re: How to iterate from ts3Get

Anonymous — Tue, 28 Apr 2020 08:10:33 GMT

Hi
Use tS3List to iterate each file, the job looks like:
tS3List--iterate--tS3Get--oncomponentok--tFileInputJson....balabala

From your image, I see you get all files to local system first, and then iterate each file and read the file one by one.

Regards
Shong

Re: How to iterate from ts3Get

manodwhb — Tue, 28 Apr 2020 08:13:06 GMT

@vvengala ,you use on component ok.

Re: How to iterate from ts3Get

Anonymous — Tue, 28 Apr 2020 08:45:49 GMT

HI @shong ,

I have tried it but getting the below error after processing first file:

Connection pool shut down
Exception in component tFileList_1 (S3_Multiple_Files_Connection)
java.lang.RuntimeException: No file found in directory C:/Users/Vivek/Desktop/AWS/JSON/S3Connection/IPfiles
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tFileList_1Process(S3_Multiple_Files_Connection.java:3139)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tS3List_1Process(S3_Multiple_Files_Connection.java:3636)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tS3BucketExist_1Process(S3_Multiple_Files_Connection.java:3362)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tS3Connection_1Process(S3_Multiple_Files_Connection.java:3846)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.runJobInTOS(S3_Multiple_Files_Connection.java:4076)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.main(S3_Multiple_Files_Connection.java:3925)
[statistics] disconnected

Re: How to iterate from ts3Get

Anonymous — Tue, 28 Apr 2020 09:10:15 GMT

Take a look at my job design in previous post, I don't use tFileList.
or you want to get all files to local system first?
In your job, you need to use onSubjobOK connector to control the execution order, ensure the files are downloaded to local system before they are used.
tS3List....
|onsubjobok
tFileList....

Regards
Shong

Re: How to iterate from ts3Get

Anonymous — Tue, 28 Apr 2020 09:14:38 GMT

@shong , I want to get one file and process it, then get the other file and process it. It should repeat for all the files.

I have removed the ts3close connection, which runs on tfilelist subjob ok. it is working fine now.

Re: How to iterate from ts3Get

Anonymous — Tue, 28 Apr 2020 09:17:45 GMT

OK, glad to see you get it working!