Qlik Community

QlikView Scripting

Discussion Board for collaboration on QlikView Scripting.

Announcements

Breathe easy -- you now have more time to plan your next steps with Qlik!
QlikView 11.2 Extended Support is now valid through December 31, 2020. Click here for more information.

twa
Not applicable

Which is faster for incremental loads: Partial Reload or Full Reload

I have a QVW that serves as the "data model" for a downstream visualization app (binary loaded) in which I am pulling incremental QVX files from a datasource (e.g. from Salesforce.com). Imagine 5-10 tables in memory with 2-20 columns each and 10 - 1,000,0000 records each.

I am wondering which method will be faster and provide greater scalability as the number of rows and the number of tables in the data model increase: traditional "Reload" with the WHERE NOT EXISTS(keyField) OR a partial reload - also with WHERE NOT EXISTS(keyField).

Is this dependent on data complexity (e.g. number of columns, cardinality, numbers of incremental rows, etc.)? Or, is it simply always better to use the Reload with WHERE NOT EXISTS()?

Any experience - and especially data-backed benchmarking! - would be appreciated.

The partial reload option would require code patterned after the following:

IF IsPartialReload() <> 0 THEN

 

          RENAME TABLE Sales to SalesPrevious;

 

          Sales:

          NoConcatenate

          ADD Load

                    sales,

                    product,

                        keyField,

                    keyField as keyNew

          From IncrementalData.qvx (qvx);

 

          CONCATENATE (Sales)

          add LOAD *

          RESIDENT SalesPrevious

          WHERE Not Exists(keyNew,keyField);

          DROP Table SalesPrevious;

          DROP FIELD keyNew;

 

ELSE          // Full load

          Sales:

          Load

               sales,

               product,

               keyField

          From FullHistoryData.qvx (qvx);

 

ENDIF