tS3List Filtering file name

[rishind]

Hi ,

I am using tS3Filelist to read file from S3 Bucket.

I have 3 files with name as

a)

EMPLOYEE_USA_TYPE_FF_DDMMYYYY

b)EMPLOYEE_UK_CLASS_SS_DDMMYYY

c)EMPLOYEE_NETWORK_TYPE_DDMMYYYY

a nd b have same schema and are used by 1 job

For c we have different job.

For Job 1 (or for a and b) I am using "/PARTY_" in key prefix

and for job 2(for c) I am using "/PARTY_NETWORK_" in key prefix

So when 3 files arrive, job 2 picks a and b files also, which results in failure

How can I change key prefix for job 1 and 2 so that job 1 picks a and b

and job 2 pick c

[Anonymous]

The prefix "/PARTY_" is a prefix of "/PARTY_NETWORK_", so of course there is going to be an overlap. If you want to avoid this issue, you need to make sure that one prefix is not an exact match of the beginning of another prefix.

[Anonymous]

Sorry, I am confused here. I am not talking about filenames, I am talking about the prefixes used. One way of potentially solving this (if the prefixes cannot be changed), is to run the job that looks for "/PARTY_NETWORK_" file first. Then remove or rename those files once it has finished. Then when you run the job that looks for "/PARTY_", the other files will not be picked up.