Optimized load

[maleksafa]

Hi,

i have the below scenario that i need to solve:

got 2 qvds that i am loading to the same table: FBSales (70+ million records) and FBInventory (8 million records), the execution script is as follow:

1) Load FBSales, that takes only 1 minute to load (in optimized way)

2) Load FBInventory, that takes 9 minutes to load (in optimized way) but the thing here is that if you check the execution script window the load of this qvd will start after 8 minutes from loading the FBSales, i guess it is something related to indexing.

the only way to make the two loads optimized is if i load them in that order, first FBSales and then FBInventory, if i do it the other way around only FBinventory will be optimized but the load time will be faster than the above scenario.

i have tried to first create the table by reading the FBSales and adding a where condition like 1<>1 in a way that it will create an empty table from the FBSales qvd, then read the FBInventory in optimized way but the FBSales is read in unoptimized way.

any idea how i can solve this issue while keeping the two load optimized (considering that this was a test and the actual data will be around 150+ million records)

[maleksafa]

but that's very weird for an optimized load, because if i load the qvds the other way around, only one of them will be optimized but the overall load time will be much faster.

[tresesco]

That is probably because, qv takes more time to resturcture a table than this unoptimized load. I did not test it, still it seems that this case (loading your second your table fast) would only be faster iff the second table is a subset of the first one. Because, that would lead not restructure the first table.

[Anonymous]

Malek

What you need to do is when you create the 2 source qvd's, add the extra fields in with values of null.  Such that both your input qvd's have the same structure.

This was recommended to me by stevedark and I have used it and it works perfectly for me.

[maleksafa]

i will give it a try, but adding null fields to the second load will make it not optimized, so i guess it is a compromise to take between optimized load and this.

[Anonymous]

Malek

That was not what I meant.

What I meant was when you create the 1st source qvd add the fields as Null's that are in the 2nd qvd but not this 1st qvd.

Then when you create the 2nd source qvd add the fields as Null's that are in the 1st qvd but not this 2nd qvd.

Such that both the resulting 1st and 2nd qvd 's have identical structures.

[stevedark]

Thanks for the mention Bill.