1 Reply Latest reply: May 22, 2014 10:08 PM by Sergey Makushinsky RSS

    incremental load of files

    Steve Lord

      Hi, I have many folders with datafeed files accumulating on a daily basis and a qlikview of the contents of those files.  As the files become more numerous, the qlikview reload time increases.

       

      Presently I have the qlikview process files dated more recently with:

      Load.. From... Where NUM(<FileDateExpression>) >= NUM(Now()-2)

       

      Then it loads the qvd of previously processed data with:

      Load... From... Where NUM(<FileDateExpression>) < NUM(Now()-2)

       

      This does a nice job of having it only do the processing work on newer files, and bring the previously processed stuff from older files out of the qvd that was made previously.  HOWEVER, the qlikview is still parsing through all of the files to check inside and see if any data to load.

       

      Can someone provide me the script piece for qlikview to parse the file names themselves and not even look into the files when loading?  Would that save any more processing time in addition to time saved from not trying to process stuff in the files?

       

      The filedate expression I have in the file loads is built on the filenames themselves, so would work to identify the files in such a script to tell it which ones to parse.

        • Re: incremental load of files
          Sergey Makushinsky

          You can use variables as file names or part of file name.

          Like

           

          //load old data from qvd

          [data]:

          LOAD *

          FROM olddata.qvd (qvd);

           

          //load the new data

          FOR i=0 to 2

          Concatenate(data)

          LET vDate=Date(Now()-$(i));

          LOAD *

          FROM

          data$(vDate).xls

          (biff, embedded labels, table is Sheet1$);

          Next

          STORE data to olddata.qvd;