tMap with multiple files as Main input and multiple files as lookup

[cdhemant]

Hello,

I have a transformation job which have 3 different entities

Entity1 may have 1 to n number of csv files, each file with 1m records

Entity2 may have 1 to n number of csv files, each file with 1m records

Entity3 may have 1 to n number of csv files, each file with 1m records

I am using tMap to join this entities, where Entity1 is the Main input and Entity2, Entity3 are lookups.

However i am not able to understand how to design the flow as the input data of file may or may not match with lookup file in processing.

Also not sure how iterator or tloop can help in this case

Thanks

Hemant

[manodwhb]

@not specified not specified​ , if you have same file structures across the all files in different sets then you can try to maintain a table to have source file and lookup1 file and lookup2 file and then you can try to use tflowtoiterate those file names and then you can try to read those files and do the next operation what ever you want.

Thanks,

Manohar

[cdhemant]

Hello Manohar,

Thanks for reply.

All files of Entity1 have same structure, All files of Entity2 have same structure, All files Entity3 have same structure.

However challenge is while processiong, if one of the row of Entity1 of Entity1File0.csv does not find matching record in Entity2File0.csv and then it can generate wrong data.

Thanks

Hemant

[manodwhb]

@not specified not specified​ ,if that is the case you need to read one source file and compare the all the lookup files,it will take long time to process. check that for lookup files use file mask and check is it working.

Thanks,

Manohar