topic Re: read multiple csv filenames with timestamp in Talend Studio

read multiple csv filenames with timestamp

Anonymous — Sat, 16 Nov 2024 02:28:42 GMT

Hi,

I'm new to talend. I'm trying to work on a job for below scenario.

1.Daily multiple csv files with filename having timestamp are loaded in ftp folder.

2. I need to read each csv file and work and load it into separate table in mysql.

Example of filenames:

AAA_21042020_144530.csv

BBB_21042020_144535.csv

CCC_21042020_144538.csv

These three files have to be read using separate tfileInputDelimited option and load into respective tables.

But as the filenames have timestamp which changes daily, we cannot have wildcard option in tfileInput Delimited.

After exploring I found, we can use tfileList to iterate through multiple files and use filemask. But to read multiple files with different patterns , multiple tfilelist have to be opened.

Can someone share a better solution to this problem.

Re: read multiple csv filenames with timestamp

TRF — Sun, 10 May 2020 20:25:51 GMT

1 single tFileList connected to an empty tJava which is in turn connected by 3 if triggers to 3 tFileInputDelimited.
If conditions are based on tFileList_1_CURRENT_FILE to determine which way to go using String.contains() or String.startsWith() methods.

Re: read multiple csv filenames with timestamp

Anonymous — Tue, 12 May 2020 08:03:21 GMT

Thank you TRF for sharing the solution.

I would like to know, how do I fetch files from multiple folders into tFileList and carry out the remaining steps.

Scenario:

1. I have files placed in multiple folders, which i need to read through tFileList.

2. Then based on filename, I need to load data into corresponding table.

For point 2, the solution you provided is working. I need inputs for point 1.

Thanks.

Re: read multiple csv filenames with timestamp

martel — Tue, 12 May 2020 08:23:10 GMT

Why dont you use the tFileList with regex : _(21042020)_

and have the list only for the day 21042020 (current_date day ?)

and for each in the list order by the second party (_144530.csv) ascending call your open file csv

Re: read multiple csv filenames with timestamp

Anonymous — Fri, 15 May 2020 12:44:56 GMT

Hi,

I need solution to read files from different folders and also pick the latest files, if there are mutliple copies of files placed in a folder.