how to load multiplefiles into multiple tables and the have to load only the latest filedate only

[k526]

Hi,

I have to load 5 files with  filenames example as indi_20101010052121.csv,

                                                                            can_20101010052121.csv,

like wise .i have created a job ltfileinputdelimted--->tmap--->tmysqloutput

can u please suggest how to load only the latest files .

Thank you

[Anonymous]

It looks like your files contain a timestamp (yyyyMMddHHmmss). As such, you can order the files (tSortRow) by the filename (assuming that the file prefix indicates a different type and you want to retrieve the latest for each type). If you want to filter the files to 1 file (the latest) per type, use a tAggregateRow component, group by type (you will need to extract that using Java String utils) and return the First function for all of the columns (assuming you have sorted the data by type and date.

[k526]

tsortrow we can use for columns .but i need based upon the filename date the inputfiledelimeted has to load latest file to the database

[Anonymous · Accepted solution]

Hi,

Please follow the below steps, this might help.

1. tfilelist component and set the parameter Order by and Order action in such a way that latest file comes last.  

2. Take a iterate Link from tfilelist to tjava component.

3. Create context variable for the file name. Inside tjava set the context variable to the talend varibale ((String)globalMap.get("tFileList_1_CURRENT_FILE"));

4. Take out a Onsubjob ok from tfilelist to your existing tfileinputdelimited. use the context name for the file name.

Cheers!

Gatha

[k526]

I dont have idea

3.Create context variable for the file name. Inside tjava set the context variable to the talend variable ((String)globalMap.get("tFileList_1_CURRENT_FILE"));
4. Take out a Onsubjob ok from tfilelist to your existing tfileinputdelimited. use the context name for the file name.

can you please help me in this?

[Anonymous]

here are the screenshots on how you can do

under tfileinputdelimited component, pls configure like below Instead F:/Gatha -pls use your file path.

In tjava component configure as below

[Anonymous]

Did you actually test this before accepting it @k526? Given your description, it won't work. This will ONLY run the latest file (singular). Given your example files....

indi_20101010052121.csv

can_20101010052121.csv

.....only one file would be loaded, yet according to the timestamp they should potentially both be loaded. I assume that the time in the filename is what you want to sort by and not the actual date of the file. 

The solution I suggested assumed an understanding that you would have to use the tFileList and plug it into a tIterateToFlow. This would then allow you to follow the rest of what I suggested.

[k526]

thanks for ur suggestion.I have tried what u suggested it was working .and i have found one problem that all the 5 files are loaded into a single table like wise 5 files are loaded 5 times in every table.I need solution like for

EX:Can file has to load only in can table,

ind file has to load only in ind table.

[Anonymous]

HI,

Can you please post a screenshot of your present program?

[k526]