How to iterate from ts3Get

[Anonymous]

Scenario: I have a few files in S3 bucket, now I want to get only a single file at a time and process the file and then get another file again and process it again, this should continue for all the files present in the S3.

please find the job design.

Thanks in advance.

[Anonymous · Accepted solution]

Hi
Use tS3List to iterate each file, the job looks like:
tS3List--iterate--tS3Get--oncomponentok--tFileInputJson....balabala

From your image, I see you get all files to local system first, and then iterate each file and read the file one by one.

Regards
Shong

[manodwhb]

@vvengala ,you use on component ok.

[Anonymous]

HI @shong , 

I have tried it but getting the below error after processing first file:

Connection pool shut down
Exception in component tFileList_1 (S3_Multiple_Files_Connection)
java.lang.RuntimeException: No file found in directory C:/Users/Vivek/Desktop/AWS/JSON/S3Connection/IPfiles
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tFileList_1Process(S3_Multiple_Files_Connection.java:3139)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tS3List_1Process(S3_Multiple_Files_Connection.java:3636)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tS3BucketExist_1Process(S3_Multiple_Files_Connection.java:3362)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.tS3Connection_1Process(S3_Multiple_Files_Connection.java:3846)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.runJobInTOS(S3_Multiple_Files_Connection.java:4076)
at aws.s3_multiple_files_connection_0_1.S3_Multiple_Files_Connection.main(S3_Multiple_Files_Connection.java:3925)
[statistics] disconnected

[Anonymous]

Take a look at my job design in previous post, I don't use tFileList.
or you want to get all files to local system first?
In your job, you need to use onSubjobOK connector to control the execution order, ensure the files are downloaded to local system before they are used.
tS3List....
|onsubjobok
tFileList....

Regards
Shong

[Anonymous]

@shong , I want to get one file and process it, then get the other file and process it. It should repeat for all the files.

I have removed the ts3close connection, which runs on tfilelist subjob ok. it is working fine now.

[Anonymous]

OK, glad to see you get it working!