3 Replies Latest reply: Sep 21, 2017 12:30 AM by Oleg Troyansky RSS

    Increasing QVD sizes when loading and storing in a long loop

    Oleg Troyansky

      Hi everybody,

       

      I'm experiencing a strange phenomenon, and I'd like to ask if anyone had seen the same, and what can be done about it.

       

      I have a Qlik Sense script that is running over hundreds of folders (using the famous DoDir logic) and transforming thousands of Text files into QVD files.  All text files from a certain folder are stored in one QVD file.

       

      So, in a nutshell, the overly simplified logic is the following:

       

      FOR EACH ...

       

           FOR EACH ....

                Table1:

                LOAD ......

          NEXT

       

           STORE Table1 INTO .....

           drop table Table1;

       

      NEXT

       

      We noticed that the size of each QVD file goes and grows, even though the volume of data in each QVD is comparable. If the first QVD file could be approx. 20MB in size, the last QVD file was about 80MB in size. The volumes of data are comparable - in fact, the smaller QVD file in the beginning of the run contained more data than the 4-times larger QVD file from the end of the run.

       

      I have a feeling that it has to do with the size of the index that's being built. However, since I'm dropping every table immediately after storing it, I don't know what else I can do to prevent the growth of the data size.

       

      Any ideas about the reasons for this issue and possible solutions?

       

      I'd love to explore a possibility of spinning out separate tasks and loading one folder at a time, but I don't know how can I do it in Qlik Sense without the ability to pass parameters to the script.

       

      appreciate your thoughts,

       

      Oleg Troyansky