I'm getting a little frustrated about an application i'm developping (financial purposes).
I have multiple fact tables which i want to concatenate to a unique one, linked to various dimensions; tables can be grouped in 2 categories : pure financial & purchases, the biggest one is about 25 millions rows (financial).
When i reload my script, the financial ones are concatenated correctly, but as soon as QV reaches the first purchase table, there is a RAM consumption peak (from 2.5GB to 10GB, and there is max. 300000lines in each purchase table) : do you have an idea where it could come from ?
I know my concaternation aren't optimized since there are explicitly forced and tables don't have same numbers of columns; but i doubt it could come from here considering the number of lines in new tables to concat and the giant RAM consumption peak; it reminds me the behaviour of a cartesian product !
Script is attached, problem appears in tab "Faits", line 43. This app is next binary loaded by.a qvw which contains charts & other visualizations items.
I already tried to concat them differently, one by one or with intermediate concats :this is the current solution in attached file,in which i try to concat a 33millions lines table (finance) with a 300839 lines table (purchase).
In my experience, when you force concatenate, QlikView has to generate the columns that are new but then goes off and creates the links to the existing rows that have already been loaded.
I always will load the largest table first and, if I am having problems like yours, make sure that I load the fields that I know will be added later with null values:
Load
SalesID,
ProductID,
SalesValue,
Quantity,
null() as PurchaseField1,
null() as PurchaseField2
...
That way, the fields are already created and you shouldn't see the spike.
