Thanks for responding.
a. Don’t think (but not sure) that it makes a difference, since the group by is also on the smallest variable, meaning it will calculate the sum for each individual line, with values of all four dimensions in it.
b. Laptop just stops. 8Gig Mem is not enough and disk also is too small. Might be something to consider pc, but need to be certain that it will function / perform to my needs.. Looking into it.
Brinklaan 120 | Bussum | The Netherlands
IBAN NL38ABNA0556315731 | VAT (BTW) 821513941B01 | KVK 32162131
BrandBooming is a registered trade name of Silver Bullet BV
I think bill.markham is right and it will be make bigger differences how do you handle such a group by load. I would probably take this way:
- load and store as qvd 10M chunks from your source without transformations and then delete the table
- load and store as qvd each chunk with your group by and then delete the table
- load all group-by qvd's and concatenate them
This are good prerequisite for incremental loads: Incremental Load
Maybe there are further possibilities, for example to flag your "A" and "B" field if they are NULL or 0 or there are other criterias which you could filter out by a where-clause. Further do you need each key / each field in this table - maybe you could switch your fields/keys to other tables or could use other (numeric) keys or ...
By the way: a laptop with 8GB is not really a well suited tool for these amount of data
can you split the group by using KeyOne values (one KyeOne value or a group of KeyOne values)? this shouldn't change the result of the group by
I think something like
create a table with some KeyOne values
read from qvd where exists KeyOne values and group by
store in qvd
at the end you have some qvd with the group by result