how to load multiplefiles into multiple tables and the have to load only the latest filedate only

[k526]

Hi,

I have to load 5 files with  filenames example as indi_20101010052121.csv,

                                                                            can_20101010052121.csv,

like wise .i have created a job ltfileinputdelimted--->tmap--->tmysqloutput

can u please suggest how to load only the latest files .

Thank you

[Anonymous]

Hi @k526,

I am not sure how the above screen shot works. 

from my solution  (Having a tfilelist, tjava and your old program)

1. In tfilelist you can specifiy the file mask. Please set a file mask as indi_*.csv

2. Please use a set of tfilelist and tjava component for each file you have with new file mask to filter the respective files.

Cheers!

Gatha

[Anonymous]

@k526 you need to use a tAggregateRow after the tSortRow and group by the filename prefix (before the datestamp) then select the FIRST function for all of the columns. Then add a tMap after the tAggregateRow and send the data for one filetype (by file prefix) in one direction and the other filetype in the other direction.

After your tMap you should connect to a tFlowToIterate, then connect that to your file. If you DB has the same schema for all files, you could parameterise the table name in the DB component and then for every file that is iterated through, you would be sending your data to a different DB table.

[k526]

Thank you so much gatha.this is working fine

[Anonymous]

hey,

same scenario.. only difference is my file is on AWS S3. i need to load files from S3 to table by ordering them on file timestamp.. i.e., old file timestamp will be loaded first.. then the next one will be loaded in the same table..

concern is, ts3list have no option for ordering. and i can not use tfilelist as tfilelist only works on premise level.

 please help.

[Anonymous]

Hello

There is no recursive function in tS3List component.
Have you tried to check out "List all buckets objects" option and enter the prefix of files to be listed? In this way, tS3list will list all the files on the S3 server.

Best regards

Sabrina

[Anonymous]

@gatha_vdm  Thanks for the quick response. I will try the given steps to get the data in multiple output tables.