topic Re: tS3List Filtering file name in Talend Studio

tS3List Filtering file name

rishind — Thu, 08 Dec 2022 14:35:21 GMT

Hi ,

I am using tS3Filelist to read file from S3 Bucket.

I have 3 files with name as

EMPLOYEE_USA_TYPE_FF_DDMMYYYY

b)EMPLOYEE_UK_CLASS_SS_DDMMYYY

c)EMPLOYEE_NETWORK_TYPE_DDMMYYYY

a nd b have same schema and are used by 1 job

For c we have different job.

For Job 1 (or for a and b) I am using "/PARTY_" in key prefix

and for job 2(for c) I am using "/PARTY_NETWORK_" in key prefix

So when 3 files arrive, job 2 picks a and b files also, which results in failure

How can I change key prefix for job 1 and 2 so that job 1 picks a and b

and job 2 pick c

Re: tS3List Filtering file name

Anonymous — Thu, 08 Dec 2022 15:39:00 GMT

The prefix "/PARTY_" is a prefix of "/PARTY_NETWORK_", so of course there is going to be an overlap. If you want to avoid this issue, you need to make sure that one prefix is not an exact match of the beginning of another prefix.

Re: tS3List Filtering file name

Anonymous — Thu, 15 Dec 2022 11:58:05 GMT

Sorry, I am confused here. I am not talking about filenames, I am talking about the prefixes used. One way of potentially solving this (if the prefixes cannot be changed), is to run the job that looks for "/PARTY_NETWORK_" file first. Then remove or rename those files once it has finished. Then when you run the job that looks for "/PARTY_", the other files will not be picked up.