'Where' Performance reading from QVD;

[#1]

Hello All!

Need some advice on the best way to approach this,

I'm trying to load data from a (Optimised) QVD,

I have around 15-16 individual load statements with separate 'Where' clauses, all concatenated together.

As the load times are significantly slower than i expected - please can anyone offer any advice on how to improve performance.

Example

Controls:

LOAD

'Opp' as [Type],

id,

Date

FROM Ticket.QVD (Qvd)

WHERE

[Task 1 Due] <= $(vToday);

Concatenate

LOAD

'Achieved' as [Type],

id,

Date

FROM Ticket.QVD (Qvd)

WHERE

[Task 1 Due] <= $(vToday) AND

[Task 1 Due] >= [Task 1 Done];

....................

[vincent_ardiet]

Hi,

Have you try to use a resident table ?

First load Ticket.QVD with no filter then perform several resident load.

Regards,

Vincent

[#3]

Hey,

Yes, i have tried but as we have a very large data set, holding that much data in memory (resident) also was having difficulties.

Thanks

[vincent_ardiet]

Ouch !

And if you try to mutualize some loading, for example something like :

Controls:

LOAD

'Opp' as [Type],

id,

Date,

[Task 1 Due],

[Task 1 Done]

FROM Ticket.QVD (Qvd)

WHERE

[Task 1 Due] <= $(vToday);

Concatenate

LOAD

'Achieved' as [Type],

id,

Date,

[Task 1 Due],

[Task 1 Done]

RESIDENT Controls

WHERE

[Task 1 Due] <= $(vToday) AND

[Task 1 Due] >= [Task 1 Done];

Then you can drop the unused columns.

Regards,

Vincent