topic Re: Copy most recent file from one directory to another in Talend Studio

Copy most recent file from one directory to another

Anonymous — Mon, 06 Mar 2017 12:03:44 GMT

Hi everyone.
I am trying to build a job in open studio that copies a .txt file from one location to another location. There will be a number of .txt files in the directory, but I only want to copy across the most recent one.
for example
file1.txt
file2.txt
file3.txt
I am trying to do this based on the date modified properties.
I have looked through a few posts on here but have not managed to get any solution to work.
Would anyone be able to point me in the right direction, or even better if you have any print screen examples I could follow?
Many thanks in advance!

Re: Copy most recent file from one directory to another

TRF — Mon, 06 Mar 2017 13:27:37 GMT

Hi,
Did you try tFileList ordered by modification date desc tFileProperties (mtime and mtime_string fields give you the last modification date and time). You may also decide to break the iterate loop or jump over the undesired rows (see this post https://community.talend.com/t5/Design-and-Development/Break-From-Iteration/td-p/99207).
Regards,
TRF

Re: Copy most recent file from one directory to another

Anonymous — Mon, 06 Mar 2017 13:42:30 GMT

I have tried the solution in the attached image.
I keep getting mtime = null in the tlog though. I am clearly doing something wrong?

Re: Copy most recent file from one directory to another

TRF — Mon, 06 Mar 2017 14:21:35 GMT

Use "((String)globalMap.get("tFileList_2_CURRENT_FILEPATH"))" in tFileProperties and put a tLogRow just after this component.
I've just tested on my environment. It works like a charme.

.---------------------------------------------------------------------+------------------------------------------------+--------------------+-----------+----+-------------+----------------------------.
|                                                                                               tLogRow_1                                                                                               |
|=--------------------------------------------------------------------+------------------------------------------------+--------------------+-----------+----+-------------+---------------------------=|
|abs_path                                                             |dirname                                         |basename            |mode_string|size|mtime        |mtime_string                |
|=--------------------------------------------------------------------+------------------------------------------------+--------------------+-----------+----+-------------+---------------------------=|
|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0170301.csv|xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|xxxxxxxx20170301.csv|rw         |103 |1488442149885|Thu Mar 02 09:09:09 CET 2017|
'---------------------------------------------------------------------+------------------------------------------------+--------------------+-----------+----+-------------+----------------------------'

Regards,
TRF

Re: Copy most recent file from one directory to another

Anonymous — Mon, 06 Mar 2017 15:10:18 GMT

Yes, This is working and it is now writing the most recent file properties to a tlog, which is great.
How do I use those details in the tlog to then copy the file across to a new location?

Re: Copy most recent file from one directory to another

TRF — Mon, 06 Mar 2017 16:56:34 GMT

Just have to configure a tFileCopy with the required informations

Re: Copy most recent file from one directory to another

Anonymous — Tue, 07 Mar 2017 08:26:20 GMT

Thank you TRF.
This is the part I am struggling with. The tlogrow only has a error message or a number of lines option to feed into the tfilecopy. Using the current filepath from the tfilelist only copies across the first file it comes to in the original directory and not the most recent file.
Apologies if this is an easy task, I have only recently picked up talend again and I am trying to get up to speed.
Regards,

Re: Copy most recent file from one directory to another

TRF — Tue, 07 Mar 2017 10:09:24 GMT

Hi,
This is not normal to have only 1 field except if tBufferInput_1 schema is not aligned.
Did you check it?

Re: Copy most recent file from one directory to another

Anonymous — Tue, 07 Mar 2017 10:49:38 GMT

Apologies I did make some changes to the job as I was not getting the tbuffer to work.
This is a print screen of my job. Up to the tlogrow everything appears to work as expected in that it outputs the file properties of the most recent file to the tlogrow.
You can see the options I have on the left in terms of what to use in the tfilecopy.

Re: Copy most recent file from one directory to another

TRF — Tue, 07 Mar 2017 11:30:37 GMT

OK, it doesn't work like this.
Click on tLogRow then edit schema. Here are the available fields.
So in tFileCopy, you can use abs_path to indicate which file you want to copy.

Re: Copy most recent file from one directory to another

Anonymous — Tue, 07 Mar 2017 11:44:07 GMT

Thanks again for your response.
I can see the abs_path in the schema of the tlogrow, but how do I use that in the tfilecopy to indicate the file to use?

Re: Copy most recent file from one directory to another

TRF — Tue, 07 Mar 2017 13:08:59 GMT

Well, for tFileCopy try row7.abs_path.
Should work.

Re: Copy most recent file from one directory to another

Anonymous — Tue, 07 Mar 2017 13:50:07 GMT

You my friend are an absolute star!
Do you happen to have any guidance to explain the row7 part? It is working like a charm.
Many thanks!

Re: Copy most recent file from one directory to another

TRF — Tue, 07 Mar 2017 14:08:33 GMT

Thank's Richard, it's a pleasure for me to help people as often as I can.
If you click the red arrow after tLogRow (row7), details are displayed on the component tab.
Just refere to flowname.fieldname to use them as soon as they have a value.
Carefull, if you refere to flowname.fieldname in a subjob after the one where this flow is defined, you'll get the las value for this field.
Regards,
TRF

Re: Copy most recent file from one directory to another

Anonymous — Tue, 07 Mar 2017 14:30:43 GMT

Great, very helpful!