Data Load Editor - Modeling

[naamah]

Hi

I want to load a sample of the data just to verify the validity of joins and to avoid a long loading time. I filter one of the QVD files and expect the others joined to it to be filtered accordingly. But in practice the other QVD files are fully loaded and the loading time is long. Is it not possible to perform a partial load? also, does the position in the script of the filtered table make any difference ?

[Vegar]

There is a 🐞 Debug load option in Qlik Sense load editor that can limit your load to X number of rows. 

See: https://help.qlik.com/en-US/sense/February2023/Subsystems/Hub/Content/Sense_Hub/LoadData/debug-data-...

[naamah]

there are some problems with your suggestion:

1. i need to press "start" every step

2. does it  load 100(for example) records randomly for each QVD? if yes than it does not necessarily match with the joins

3. it still takes long time ,  i think it goes for full load and filter 100 after it.

[marcus_sommer]

You may apply a FIRST statement to each load, like:

t: first 1000 load * from source.qvd (qvd);

but it will always load the same records unless the source record-order changed. If you want to load random records you will need an appropriate where-clause maybe something in this way:

where mod(recno(), $(var1)) = $(var2);

and the variables are maybe something like: ceil(rand() * 1000)

Beside of this I suggest to re-think your approach because optimized qvd-loads don't need much time and if the join caused a heavy processing there are probably too much common fields between the tables and/or too much records - as it would be sensible to use a join.

Especially in regard to your mentioned target to validate the results indicate that the data-quality might be problematic so that other approaches may more suitable, for example replacing the joins with mapping or not merging the data on a horizontal level else vertically which means to concatenate the tables.