Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Dear All,
I am working with millions of rows for the first time and wonder what is the most efficient approach.
On the same dataset, a colleague created a script utilizing preceding load straight into the front-end qvw with the following idea:
I tried to use an approach with incremental load using separate Qlik-tables for each DW-table: this took a lot longer than my colleague's script.
*** at this point I have two tables and qvds with the whole dataset ***
4. Create an inner join between the two tables
5. Store this table in a qvd
6. Load this table from qvd and
*calculate Delta-field based on the difference of 2 fields
*use an if clause to create a Status-field based on the values of Delta-field
Sorry if this is ambiguously written, what I am basically asking is this:
In general an incremental load with qvd's should be faster than sql-loadings against a database.
But it will be depend on various things how much faster it could be and the efforts which are needed for it.
For example, is the database quite fast and the limiting factors are mainly the database-driver and/or the network it could be that the inner join + the few transformations (status + truncating) are faster performed as the same rawdata without any transforming are transferred to qlik.
In this case you couldn't get this step faster with qlik but you could load this joined table with an incremental approach - and I think this is the essential point here - by an incremental loading you need not only load the rawdata incrementally you must also load the transformation-results incremental and by this building a multi-staging load-approach.
For your case it might be a bit expensive to do this but if load-performance is a bottleneck or you could use thes qvd's in other cases it might be worth.
- Marcus