topic Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder in Talend Studio

Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Anonymous — Sat, 16 Nov 2024 07:33:11 GMT

Hello,

On my ftp location, I have folders: Alpha, Beta, Chroma, Delta which in turn have folders, Alpha1, Alpha2, Alpha3; Beta1, Beta2, Beta3; Chroma1, Chroma2, Chroma3; Delta1, Delta2, Delta3 respectively. Each of these folders have .csv files with names, Data_date.csv (ex. Data_01_01_2018.csv). The file names are common along folders. I have some of these files on my in folders by the same names as FTP location. I need to compare both, get a list of files that are not present and pick those files to pc storage. Can someone suggest me how to do this?

In future, I also need to have it so that there are no files on PC but only the list of files that were before and whatever filename isn't on the filelist, has to get picked.

Anyone here to help?

Thanks in advance.

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

fdenis — Mon, 01 Oct 2018 13:02:52 GMT

use tFtpFileList --iterate--> tIterateToFlow --> tMap --> ListOf nonMatched File
use tFileList --iterate--> tIterateToFlow --------^

ftp: tIterateToFlow use:
(String)globlaMap.get("fFTPFileList_1_CURRENT_FILEPATH")
(String)globlaMap.get("fFTPFileList_1_CURRENT_FILE")

filelist: tIterateToFlow use:
(String)globlaMap.get("fFileList_1_CURRENT_FILEPATH")
(String)globlaMap.get("fFileList_1_CURRENT_FILE")
good luck

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Anonymous — Mon, 01 Oct 2018 15:11:03 GMT

@fdenis, how about the folders? This approach works if there's one folder. Not many.

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

fdenis — Mon, 01 Oct 2018 15:23:30 GMT

on tfilelist you can include subdirectories
on tFtpfilelist you have to include the good fileMask to run throw directories.

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Anonymous — Mon, 01 Oct 2018 15:33:56 GMT

@fdenis, good file mask? They are csv files and I'm using "*.csv" as filemask. What is it I'm missing?

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

fdenis — Mon, 01 Oct 2018 15:52:43 GMT

something like "*\\*.csv"
or "**\\*.csv"
or regexp.
if it's not working you can create a subjob calling him self on folder, llisting files when files.

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Anonymous — Mon, 01 Oct 2018 16:26:34 GMT

@fdenis, not working. The file names are same across folders. So, that's a problem.

I'm trying to use another approach.

tFileInputDelimited (with names of folders - Alpha, Beta, Chroma, Delta) -> tFlowtoIterate -> tLoop -> tJava

(converting folder names to a context output)-> tFileInputDelimited (with names of subfolders)-> tFlowtoIterate -> tLoop -> tJava (converting subfolder names to context output) -> tFTPFileList -> tIteratetoFlow -> tMap (Connected to a tFileList and tIteratetoFlow) -> tFlowtoIterate -> tFTPGet

I'm giving the names of folders and subfolders and using a loop element to run it. But it doesn't work. Can anyone help?

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Anonymous — Mon, 01 Oct 2018 16:31:40 GMT

@TRF, could you help may be?

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

fdenis — Tue, 02 Oct 2018 08:52:57 GMT

It was working on an other project.
do not use CURRENT_FILE who is only filename. use CURRENT_FILEPATH who get filename and path

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

fdenis — Tue, 02 Oct 2018 08:56:01 GMT

be careful: context are mead to be job or subjob parameters.

Re: Compare filelist from ftp location with files in multiple folders on pc and pick them by folder

Anonymous — Thu, 04 Oct 2018 10:00:38 GMT

@fdenis, got it. Thank you. Figured out a different way with loops.

Thanks a lot for help.