topic File List - File properties in Talend Studio

File List - File properties

Anonymous — Sat, 16 Nov 2024 14:25:41 GMT

In a pre set directory loop through the files and delete those that are older than 2 minutes.

Processing the contents of a directory was done using the tFileList step. However, it seems that it only retrieves the File Path of a file and not any of the other properties associated with a file like Size, Created Date, Last Changed etc.
Searched the help file, forum and tutorials for the keywords file, time and date. Sadly to no avail.
Now, I probably think that this can be done within TOS, but will require some custom code, ie. Java Step but I was wondering if there was a different way of doing it. Sadly I don't have the programming skills to create this kind of functionality.
Any help will be appreciated.
Regards,

Re: File List - File properties

Anonymous — Sat, 15 Mar 2008 22:15:26 GMT

Hi,
you can use the following "hack" to get more information about the file in a tJava component after a tFileList. Change variablenames if necessary.

System.out.print(files_tFileList_1.getName() + ": ");
System.out.print(files_tFileList_1.length() + "\n");

Hack, because I use a internal variable of tFileList, this may be changed in a future version.
Actual there is no Java (standard) method to get the create time of a file.
Bye
Volker

Re: File List - File properties

Anonymous — Sun, 16 Mar 2008 15:47:13 GMT

I think you can be interested in 984 (one year old and suspended but can be reactivated). I thought creating a specific component that would output all data (filemtime, size, owner, ...) into a data flow (a row link) and you suggest to have it in tFileList global variables.
I do still prefer the dedicated tFileProperties dedicated component because you can be interested in file properties outside tFileList. Imagin you have your file list in a database table.
I encourage you to add a note in this feature request and monitor it so that we're aware some users need it.

Re: File List - File properties

Anonymous — Tue, 18 Mar 2008 12:44:47 GMT

Thanks for the suggestion; sadly the file date is the most important one at present. That said, I do feel that all the other file properties will have a use as well. In this case a message has to go out if a file is older than a particular date. Granted, it's not really ETL but since the file object is already part of TOS I had thought that the extension of the file step or addition of the properties step would be appropriate.
Kind regards,
Anne.

Re: File List - File properties

Anonymous — Tue, 18 Mar 2008 21:49:16 GMT

Sorry, my answer was not precise enough. The variable tFiles_tFileList_l is of type java.io.file. So you can use additional methods on this object. For example: long lastModifed().
More information can be found in the api: http://java.sun.com/j2se/1.5.0/docs/api/java/io/File.html

Re: File List - File properties

Anonymous — Wed, 19 Mar 2008 20:53:46 GMT

Volker, thanks for this comment. This is indeed something I might be able to process further down the road. It is indeed a work around and it will have to be documented as such. In any case, it will allow me to do a few things that are on the fringe of ETL with the same tool. Will give this a go on my other notebook in the next couple of days.
Again, thanks for helping out.
Regards,
Anne.

Re: File List - File properties

Anonymous — Sat, 22 Mar 2008 23:20:56 GMT

Just for those interested in what I ended up doing was to add a tJavaStep after the the tFileList step and added the following line of code:

globalMap.put("tFileList_1_CURRENT_FILEDATE", files_tFileList_1.lastModified());

After this I added an tIteratToFlow step with a colum called Date and filled it with the following statement:

((Long)globalMap.get("tFileList_1_CURRENT_FILEDATE"))

This will result in the Date field being populated with a long number which represents a the date in milliseconds after 1970. This date can then be used to do all sorts of comparison and calculations.
Again, thanks for responding so quickly and for all the useful information you have provided me.