topic Re: Need to set Date parameter/context variable for directory to s3 bucket in Talend Studio

Need to set Date parameter/context variable for directory to s3 bucket

NNot_defined1674577304 — Fri, 15 Nov 2024 21:42:43 GMT

Hi , Trying to create a job that would copy files from source folder to s3 bucket location. They copy file job is complete but it’s copying all files. I need to have a selection of dates like startDate and endDate in Tac so It can auto import files of chosen dates .Please suggest

Re: Need to set Date parameter/context variable for directory to s3 bucket

Anonymous — Thu, 08 Jun 2023 04:10:37 GMT

Are these files on local folder where the job is executed? or copy the files from one S3 bucket to another bucket?

Regards

Shong

Re: Need to set Date parameter/context variable for directory to s3 bucket

NNot_defined1674577304 — Thu, 08 Jun 2023 04:12:35 GMT

Not local . they are in a source path will land to s3 bucket

Re: Need to set Date parameter/context variable for directory to s3 bucket

Anonymous — Thu, 08 Jun 2023 04:43:54 GMT

If the files are on S3 bucket, you can tS3List to iterate each files, there is a global variable eg: ((java.util.Date)globalMap.get("tS3List_1_CURRENT_LASTMODIFIED")) that stores the last modified date of the file, consider using this global variable to filter the files, and only copy these selected files to another S3 bucket.

Re: Need to set Date parameter/context variable for directory to s3 bucket

NNot_defined1674577304 — Thu, 08 Jun 2023 05:04:38 GMT

Where do I use this variable ? In tfilelist filemask? Do I have to ad any contexts ? I

Re: Need to set Date parameter/context variable for directory to s3 bucket

NNot_defined1674577304 — Thu, 08 Jun 2023 05:10:33 GMT

Where do I use this variable ? In tfilelist filemask? Do I have to ad any contexts ? Also I need to have selection of any date range

Re: Need to set Date parameter/context variable for directory to s3 bucket

Anonymous — Thu, 08 Jun 2023 05:47:18 GMT

You can use this variable on a tJava and check if the date is between a date range. eg:

tS3list--iterate--tJava1---runIf--tS3Copy--oncomponentok--tJava2

on tJava1:

java.util.Date currentLastModifiedDate=((java.util.Date)globalMap.get("tS3List_1_CURRENT_LASTMODIFIED"));

if(currentLastModifiedDate.getTime()>=context.startDate.getTime()&&currentLastModifiedDate.getTime()<=context.endDate.getTime()){

context.isCopy=true;

}

// startDate and endDate are context variables, Date type.

// isCopy is a context variable, boolean type, the default value is false;

Set the condition of runIf connector as:

context.isCopy

on tJava2:

context.isCopy=false;