topic Moving the files based on file pattern in Talend Studio

Moving the files based on file pattern

Anonymous — Thu, 05 Oct 2017 12:48:57 GMT

Hi Team,

I want create job below logoc.

FIRST RUN.

1) One common directory(i.e C:/Landing)

files in C:/Landing:

AAA_2017_0610_024629
AAA_2017_0611_024623
AAA_2017_0612_024622
BBB_2017_0612_024300
CCC_2017_0612_024300

2) Move the files to specific directory based on starting pattern. like AAA, BBB and CCC.

Move C:/Landing to C:/AAA
Move C:/Landing to C:/BBB
Move C:/Landing to C:/CCC

Note:
1) Move one file at time to specific directory(Like AAA) for each run.(i will secedule job every 5 mins).
2) Once file is moved to specific direcory delete the files from C:/Landing directory.
3) Move is done based old dates not latest date. ex: move files 10th , 11th and 12th date order.(NOTE: move one file at time for each run).

Second RUN.

same process, only it will move next file if mutiple file exist based on dates.

fisrt run:
move AAA_2017_0610_024629 to c:/AAA
move BBB_2017_0612_024300 to c:/BBB
move CCC_2017_0612_024300 to c:/CCC

second run:
AAA_2017_0611_024623 to C:/AAA

etc.

Kindly help on above scanario.

Thanks

Shree

Re: Moving the files based on file pattern

Anonymous — Thu, 05 Oct 2017 13:10:56 GMT

Hi @bshree88,

You need to stop calling your questions "scanario". You need to give a decent description so that other people can make use of the answers you get. You have also asked for someone to tell you how to create the whole job again. If you are new to Talend, that is great. But you (or your company) need to give yourself time to learn the product, since it looks like you are trying to deliver functionality to either your company or a client by asking us to do it for you. That is not really good for you, us or Talend.

However, I have a job that kind of does most of this and I have quickly changed it to cover most of your requirements. You will need to tweak it to cover all of your requirements. This is a very simple job and simply uses the Java substring method to get the folder name and then uses a tFileCopy to copy and delete the file. The tFileList is used to monitor the folder and you can set the tFileList to do this in a particular order.....I have not done that.

You will need Talend v621+ to use this job

FileMove.zip

Re: Moving the files based on file pattern

Anonymous — Thu, 05 Oct 2017 13:19:25 GMT

Hi Richard,

Yes, i will take care your inputs.

Yes, I am learning and delivering the functionality. i am trying my side as well.

Thanks once again for direction with others points.

Regards

shridhar

Re: Moving the files based on file pattern

Anonymous — Thu, 05 Oct 2017 13:29:21 GMT

Hi Richard,

I have getting error. Please find the attached file.

Are you using any routines here.

Thanks

Shree

error.PNG

Re: Moving the files based on file pattern

Anonymous — Thu, 05 Oct 2017 13:36:03 GMT

Hi Richard,

I got where is a issue, the issue with jvm and talend version. i have created same job in talend version 6.4.

it's working fine.

Thanks

shree

Re: Moving the files based on file pattern

Anonymous — Fri, 06 Oct 2017 05:31:40 GMT

Hi Richard,

As per jobs. below points are my conversations.

1) It's moving all files for single run if multiple files with same pattern.

ex:

AAA_2017_0610_024629
AAA_2017_0611_024623

2) In requirement, creation of destination folders is not based on patter, it could be C:/AAA or C:/test/files etc.

3) Need to move single file for single run if only if multiple file have same patter.(in order)

Kindly give me idea/suggestion. i will work around.

Thanks for your time and support.

Regards

Shree

Re: Moving the files based on file pattern

Anonymous — Fri, 06 Oct 2017 12:16:56 GMT

@bshree88 the job is set to look at the pattern at the start of the filename, create a folder situated on the C:\ drive (if the folder does not exist), and adds the file to that folder.

Your initial question seemed to suggest that the pattern dictated the folder the file would be in? Has this changed? If you are talking about the path of the folder, you can edit the job using a context variable which can be changed quite easily. But the pattern at the front of the file must match the folder, or how do you know where the file goes?

The job will move every file in the input folder. Why would you want to limit it to one file at a time? This can be done, and would require just taking the first file returned by the tFileList (which can be set to find files in date order). An easy way of doing this is to store the files found n memory and only use the first file. There are other ways.

Re: Moving the files based on file pattern

Anonymous — Fri, 06 Oct 2017 12:19:51 GMT

The reason my job failed is that I have the implicit context load set for my project. There are parameters set which will not work on your machine. Just switch off the implicit context variables in your job settings.

Re: Moving the files based on file pattern

Anonymous — Mon, 09 Oct 2017 04:48:32 GMT

Sure Richard.

Thanks

Shree

Re: Moving the files based on file pattern

Anonymous — Mon, 09 Oct 2017 05:05:59 GMT

Hi Richard,

Please find the below are my answer.

Ans: Destination folder is anything. only i want to move the files based on patter to any destination folder(Not specific).

The job will move every file in the input folder. Why would you want to limit it to one file at a time?

Ans: Reason is i want to process old date file first to new date files one by one(Each run). need to run every 10 min(frequency is depends).

This can be done, and would require just taking the first file returned by the tFileList (which can be set to find files in date order).

Ans: But i want to set based file date and date is coming with file name. i want to substring the date order by date, pick old date and move to destination folder.

An easy way of doing this is to store the files found n memory and only use the first file. There are other ways.

Ans: How to achive 3rd one. storing in meory and move single old date file to destination folder.

Thaks for your inputs.

PFA my job which created. i am facing how to move single file to destination if mutiple file exist with same pattern based on file dates.

Regards

Shree

Job_design_5.PNG
Job_design_1.PNG
Job_design_2.PNG
Job_design_3_tjava_code.PNG
Job_design_4.PNG

Re: Moving the files based on file pattern

Anonymous — Mon, 09 Oct 2017 08:19:35 GMT

@bshree88 wrote:

Hi Richard,

Please find the below are my answer.

Your initial question seemed to suggest that the pattern dictated the folder the file would be in? Has this changed? If you are talking about the path of the folder, you can edit the job using a context variable which can be changed quite easily. But the pattern at the front of the file must match the folder, or how do you know where the file goes?

Ans: Destination folder is anything. only i want to move the files based on patter to any destination folder(Not specific).

OK, this can be achieved using a Context variable as the base destination folder rather than hard coding it. You can set the Context variable at runtime.

The job will move every file in the input folder. Why would you want to limit it to one file at a time?

Ans: Reason is i want to process old date file first to new date files one by one(Each run). need to run every 10 min(frequency is depends).

I see. If you load your tFileList values using an iterate to a tFixedFlowInput, you can then store them in a tHashOutput component (Google this if you are not sure). You can then use a bit of Java to get the date from the file and then order by the date. Then it is just a case of using only the first record.

This can be done, and would require just taking the first file returned by the tFileList (which can be set to find files in date order).

Ans: But i want to set based file date and date is coming with file name. i want to substring the date order by date, pick old date and move to destination folder.

An easy way of doing this is to store the files found n memory and only use the first file. There are other ways.

Ans: How to achive 3rd one. storing in meory and move single old date file to destination folder.

Thaks for your inputs.

PFA my job which created. i am facing how to move single file to destination if mutiple file exist with same pattern based on file dates.

I am not entirely sure what you mean here. Are you saying that files have the same date and the same folder path pattern? If so, you need to find another way of sorting (an additional way to just date) and do the same as described above.

Regards

Shree

Re: Moving the files based on file pattern

Anonymous — Tue, 10 Oct 2017 06:57:17 GMT

Hi Richard,

Please find the below are my answer.

Ans: Destination folder is anything. only i want to move the files based on patter to any destination folder(Not specific).

OK, this can be achieved using a Context variable as the base destination folder rather than hard coding it. You can set the Context variable at runtime.

Fine..

The job will move every file in the input folder. Why would you want to limit it to one file at a time?

Ans: Reason is i want to process old date file first to new date files one by one(Each run). need to run every 10 min(frequency is depends).

I see. If you load your tFileList values using an iterate to a tFixedFlowInput, you can then store them in a tHashOutput component (Google this if you are not sure). You can then use a bit of Java to get the date from the file and then order by the date. Then it is just a case of using only the first record.

Fine..

This can be done, and would require just taking the first file returned by the tFileList (which can be set to find files in date order).

Ans: But i want to set based file date and date is coming with file name. i want to substring the date order by date, pick old date and move to destination folder.

An easy way of doing this is to store the files found n memory and only use the first file. There are other ways.

Ans: How to achive 3rd one. storing in meory and move single old date file to destination folder.

Thaks for your inputs.

PFA my job which created. i am facing how to move single file to destination if mutiple file exist with same pattern based on file dates.

I am not entirely sure what you mean here. Are you saying that files have the same date and the same folder path pattern? If so, you need to find another way of sorting (an additional way to just date) and do the same as described above.

Here, Below is example same if

if files start with AAA move to C:/TEST/ and that only one file, after move replace from source direcctory. ie. AAA_2017_0612_024622(old date)

AAA_2017_0610_024629
AAA_2017_0611_024623
AAA_2017_0612_024622

I work on as mentison above points.

Regards

Shree

Re: Moving the files based on file pattern

Tasfiahm — Tue, 14 Jul 2020 16:53:01 GMT

Hi Richard,

Can you please re-share the sample job link. It will be a great help.

Thanks,