3 Replies Latest reply: Dec 23, 2013 10:14 AM by Boris Kacevich RSS

    multiple files load from multiple folders

    Boris Kacevich

      Hi all,

       

      I'm building an extract script for my app - the source Data is CSV files from a certain path in the system.

      For each run of the system a new folder is created, in this folder there are 6-7 files, each representing a different table.

      the folder name is the RUN_ID of this run.

      the meta data of the run is an analytics table - in it I have all RUN_ID's and run dates.

       

      I need to load files into existing QVDs. the high level algorithm for the load is known.

      What I need is a way to know from the analytics table, which Runs do I need to load from the file system.

      then, I need to loop through the root folder but only through the RUN_ID folders who are relevant for this load.

       

      1) How is the save of a list of needed RUN_IDs is made? (the watermark should be date or run_id)

      2) How is a loop through a root folder is made when we need to check the sub folders (the RUN_IDs) and stop when we finish with the list     we got from the first question.

       

       

      Thanks,

       

      Boris