topic Re: [resolved] tFileCopy and tFileDelete in Talend Studio

[resolved] tFileCopy and tFileDelete

Anonymous — Sat, 16 Nov 2024 12:30:23 GMT

Hello,
I try to copy a whole directory (or more) and delete it after.
To do this, i use a tFileCopy component and a tFileDelete component. The problem is when i want to do this on large file (>150Mo) it doesn?t delete every file in the directory.
I think that the problem comes from the tFileCopy. It doesn?t close the file.
Have you any suggestion ?
Regards

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 09:28:17 GMT

preatorien,
Are you trying to copy files from your source folder to another folder and delete the files in the source folder after they are copied?
If so you can use the delete in tFileCopy by enabling "Remove Source File".
If not is the filemask correct for all the files in the folder? (in your tFileList2 component)
You can try running both steps seperately to look if that makes a difference but I don't think so when using OnSubJobok.

Regards

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 09:49:22 GMT

Hi mpa,

Yes, i?m trying to copy files from source folder to another folder and delete those from the source folder after they are copied.
I can?t use "Remove Source File" actually because i copy the whole directory not just a file.
My tFileList_2 does the same iteration that the first one (with same Filemask). And, it works with little file (under 100Mo).
I think the problem comes from tFileCopy because it doesn?t close the file after the copy.
I?ve tried something with a tSleep, to see if i could delete manually the file after the tCopy and it doesn?t work, the file is locked until the end of the job. (tFileList --> tFileCopu --> tSleep try to delete files or folder manually, it doesn?t work for big file).
Thanks for your answer.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 10:09:07 GMT

preatorien,
Ok I didn't now there was an option "Copy a directory".
Have u tried running both steps seperately? By Disabling the Delete step and running the first part, when the job is complete run the 2nd step to delete the files by disabling the first two components. If this works it would mean that you are right about the locking part. If it doesn't work it's something else.
Regards

Re: [resolved] tFileCopy and tFileDelete

janhess — Thu, 15 Dec 2011 10:10:14 GMT

Your tFileList doesn't include subdirectories so you can check the remove source file in the tFileCopy.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 10:19:24 GMT

mpa --> When i run the both steps seperately, it works. So, yes, i think it's a locking problem, but i don't know how to do to go around this problem. At the moment, i've tried a lot of different solution but nothings work.
Hi janhess, i will try your solution, but, i think that, if i check "include subdirectories", my filemask doesn't work anymore. Actually, i got this problem in a bigger Job, and there is some other constraint.
Thank you.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 10:26:40 GMT

Do you have multiple directies? Or just one directory?
If it's one directory you can use file in your tFileList and tFileCopy instead of directory and do the "Remove Source File". If it's not I guess you can't do this option.
Maybe creating a second job for the second step helps. Then create a father job doing "CopyJob" -OnSubJobOk-> "DeleteJob".

Re: [resolved] tFileCopy and tFileDelete

janhess — Thu, 15 Dec 2011 10:30:06 GMT

I didn't suggest you check the subdirectories. I said that since it was unchecked you could check the Remove Source File in the tFileCopy.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 10:48:18 GMT

Sorry janhess, i haven't read your post correctly. So, if i understand, you say that, actually, i can check "remove source File" in the tFileCopy? I try to do this, but, since it's a directory and not a file, tFileCopy is not able to delete it (I got a NullPointerException).
I also try to copy and delete file by file (not the whole directory) and use the "remove source File" option. It works, but it doesn't do what i need.
mpa, i have multiple directory. i have something like this
AAA_date_time
|
---AAA_name.zip
---AAA_name
|
AAA_name.csv
And i can have multiple AAA_date_time directory. I tried to do this in to different job, i got the same problem. I think since the father job is running, the big file are locked, even if i do the copy in one child job and the delete in an other one. And i don't understand why.

Regards

Re: [resolved] tFileCopy and tFileDelete

janhess — Thu, 15 Dec 2011 11:05:31 GMT

Ah in that case have you tried checking the subdirectories box in tFileList? I think this should process everything at the file level.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 13:08:42 GMT

janhess,
I tried checking "includes subdirectories", but the problem with this solution is that, in my tFileCopy component i can't copy directories and files. Maybe, i can try to separate the copy of directories and files with two different tFileList and tFileCopy but actually, i don't know how to do this exactly (because, if i want to copy the directories first, files will be copied too, and if i copy files first, i'm not able to retrieve their former directory).

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 13:37:05 GMT

preatorien,
Put the following in your destination of your tFileCopy:
"DEST FOLDER LOCATION: What you put in your tFileCopy" + StringHandling.RIGHT(((String)globalMap.get("tFileList_1_CURRENT_FILEDIRECTORY")),StringHandling.LEN(((String)globalMap.get("tFileList_1_CURRENT_FILEDIRECTORY")))-StringHandling.LEN("E:\TestAsupprimerEnsuite\In"))+ "/" + ((String)globalMap.get("tFileList_1_CURRENT_FILE"))

This will do the following: Destination Folder + Subdirectory name + FileName.
Doing this will give you the same directory structure.

Regards

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 14:38:37 GMT

Thank you so much mpa, it works. It?s not a beautiful solution but it works.
I explain my job for those who are interested in. You can see a screenshot of the job in attachment.
tFileList_1 settings :
- Directory : ?E:/TestASupprimerEnsuite/In?
- FileListType : Directories
- FileMask : "AAA*"
Don't check the "includes subdirectories" setting.
tFileList_4 settings :
- Directory : ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
- FileListType : Files
No Filemask, Check "Includes subdirectories"
tFileCopy_1 settings :
- File Name : ((String)globalMap.get("tFileList_4_CURRENT_FILEPATH"))
- Destination directory : "E:/TestASupprimerEnsuite/Out/"+StringHandling.RIGHT(((String)globalMap.get("tFileList_4_CURRENT_FILEDIRECTORY")),(StringHandling.LEN(((String)globalMap.get("tFileList_4_CURRENT_FILEDIRECTORY")))-28))
28 is the length of the directory of tFileList_1, to do this in a beautiful way, you can declare it in context variable.
- Check "Remove source file", "Replace existing file" and "create the directory if it doesn't exist"
- don't check "Copy a directory"
The second part of the job with tFileList_2 and tFileDelete_1 is used to delete the directory (all files are already removed in the tFileCopy component).
tFileLis_2 settings : same as tFileList_1
tFileDelete_1 settings :
- Directory : ((String)globalMap.get("tFileList_2_CURRENT_FILEPATH"))
- Check "Delete Folder"
Thank you mpa.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 14:47:42 GMT

preatorien,
I don't understand why there are 2 filelists for the copy part, but if it works it's ok I guess :-).
Can I ask you a question?
How did you upload that image, I have been some time on the forum now but never understood it! I always use Regards

Re: [resolved] tFileCopy and tFileDelete

janhess — Thu, 15 Dec 2011 15:00:45 GMT

Would it be better to have the output base directory as a context variable - basedirectory and use len(context.basedirectory).
eg Destination directory : context.basedirectory+StringHandling.RIGHT(((String)globalMap.get("tFileList_4_CURRENT_FILEDIRECTORY")),(StringHandling.LEN(((String)globalMap.get("tFileList_4_CURRENT_FILEDIRECTORY")))-len(context.basedirectory)))

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 15:01:03 GMT

mpa,
I need two tFileList because all my files does not match the filemask. If i only use one of them, i lost some files. Anyway, it works. Thank you again.
To upload images, i use the "Image upload" option that is available under your message when you post a reply.
Regards

Re: [resolved] tFileCopy and tFileDelete

janhess — Thu, 15 Dec 2011 15:02:52 GMT

Click Post reply rather than use Quick post.

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 15:04:06 GMT

preatorien,
I had no idea hehe! After all this time, thanks a lot mate! Will be a lot easier for me to help people on the forums
Have a nice day,
Regards

Re: [resolved] tFileCopy and tFileDelete

Anonymous — Thu, 15 Dec 2011 15:12:56 GMT

Would it be better to have the output base directory as a context variable - basedirectory and use len(context.basedirectory).

You're right janhess, that's what i do in my real job.
Have a nice day and thanks again

Re: [resolved] tFileCopy and tFileDelete

janhess — Thu, 15 Dec 2011 16:20:38 GMT

Would this work?
context.basedirectory+((String)globalMap.get("tFileList_4_CURRENT_FILEDIRECTORY")).substring(len(context.basedirectory))