Wait for 2 unknown files to be created and then combine them

[Anonymous]

Hi guys!

I wish to create a pipeline where 2 files will be combined using tmap component, but I don't know the names of the files or the time intervals between the arrival of the new files. However, each file is expected to be found in a different folder and thus I thought of using the tWaitForFile component. Although the design seems to be ok (model 1), it seems that I cannot have 2 tWaitForFile components in the same job. Any ideas on how to solve this problem ??

P.S. I have already tried creating 2 separate jobs (model 2), in order to read each of the files separately and then store their data in a tHashInput component, while enabling the multi-threaded execution for this particular job. However, in order to continue with the process I also used on subjob ok triggers after the hashing is completed, so that the tHashOutput components should be enabled but the execution once again fails.model 1model 2

[Anonymous]

This is not a Pipeline Designer question. This is a Talend Studio question. I have moved this to an appropriate board.

[Anonymous]

Hi,

    Why don't you try file based trigger for the job? But in this case also, you need to still check the presence of second file within your job.

https://help.talend.com/reader/WEEgRoxIy_iUMxRK3bopPQ/Fy2N2XGJI3Mb57bkVMwedg

  Now, if you want to do it within the job, you should first have two subjobs to check the presence of each input file. In your model 2, you are doing it for 1 file. You should do the same process for second file too. Once you get assurance that data from both files are with you, then you can move to the subjob where you are doing the match process.

Tail note:- Please note that if the incoming data volume from files is high, you may not be able to store the entire data in Hash even if you are increasing memory parameters. You can even use a temp file instead of Hash.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved