topic Re: Slow tFilelist? in Talend Studio

Slow tFilelist?

Anonymous — Sat, 16 Nov 2024 12:26:51 GMT

Hello,
I have a job which needs the latest file from a directory with a lot of files ( over 56.000 with about 700 with the same filemask that I am searching for ).
The file I need is searchable and contains a datetimestamp in the file ( but not always from today or yesterday ).
On a local disk it runs adequate ( it finds the file in about 2 sec ) but if i try it on a windows share which has the files it takes over 40 minutes. What's wrong with it.
The filename I'm searching for is : "test." + context.customernumber + "*.txt"
with the settings sorted by date desc and then a iterate to a tjava which sets a globalvar if it's unset else it does nothing (so this way I get the latest file)
I have tried sorted by date asc and then keeping the last iteration but the time remains the same.
The setup :
Client (which runs Talend) Win7
Server (which has the files on a samba share) windows 2003 server
I am almost desperate enough to create a subjob which gets the complete filelisting unsorted and then sort them in the subjob. But I don't think this is the correct way to go.

Re: Slow tFilelist?

Anonymous — Tue, 17 Jan 2012 08:27:54 GMT

Hi
Welcome to Talend Community!
Could you explain in detail about your job logic?
I need to know what the job will do if there is a latest file. It will copy this file or move it?
Sometimes when Talend job try to handle a file(e.g. Excel) which is opend by other user, the job will wait until the file is not in use.
Or i miss some detail?
Regards,
Pedro

Re: Slow tFilelist?

Anonymous — Tue, 17 Jan 2012 12:38:13 GMT

Job logic is pretty simple ( in the test-job i created for finding the problem 😞
There are some things coming from a context.
tJava_1 :
if context property is null then the filemask should be different from when the context property is filled.
tFileList_1 :
search for all files with the filemask specified in the tJava_1 property (this takes 30 minutes in this example)
tJava_2 :
print the last record found
tFileExist_1 :
the start of the job if there is a last file.
In this example I was searching without a context property so the filemask should be : class.*
The file-specs are :
Total files : 16655
class.* files : 486
I don't see where the 30 minutes goes in this job. So there is no opening / closing of files involved. All the files on the server are closed

Re: Slow tFilelist?

Anonymous — Fri, 20 Jan 2012 10:09:10 GMT

Ok, some further investigation revealed that tFilelist with sort-options set is terribly slow.
It's about 100x faster to build a tfilelist (without sorting) -> tfileinfo -> tsortrow than to use the sorting possibiilities on the tfilelist settings.

Re: Slow tFilelist?

Anonymous — Mon, 05 Mar 2012 15:38:05 GMT

tFileList seems to have some problem when working with network paths... I have a directory containing about 2k files and tFileList freezes in spite of the very good latency time of the connection... I suppose it is a bug?

Re: Slow tFilelist?

Anonymous — Tue, 06 Mar 2012 19:03:18 GMT

@nc : What kind of network connection are you using? I was using windows-UNC paths (so I guess it uses the SMB-components).
If you are using FTP or some other network connection the problem may be somewhere else...

Re: Slow tFilelist?

Anonymous — Wed, 07 Mar 2012 10:49:49 GMT

I'm using a standard windows UNC path as of "\\serverName.domainName.local\sharedDirectory". When I open the UNC path in the windows explorer I see the list of files in a flash and I'm able to walk in each directory without any delay... In spite of the above, when I try to print the directory list with a simple job as of "tFileList->tLogRow" I have to wait many minutes...

Re: Slow tFilelist?

Anonymous — Wed, 07 Mar 2012 14:17:44 GMT

Just one note: the "order by" and "order action" setting are left on their default value.
I didn't reported well the simple job to test the behavior: it's "tFileList->tIterateToFlow->tLogRow".
Thanks,
N.

Re: Slow tFilelist?

Anonymous — Wed, 25 Jul 2018 10:45:51 GMT

It's faster to list files using system command than using tfilelist.