topic Restrict record load in QVD from table/csv in QlikView

Restrict record load in QVD from table/csv

onkarkulkarni — Wed, 25 Nov 2020 16:16:04 GMT

I have a table with 50 Milion + records in Hive. I am trying to read those records in QVD. However post 25 M , QVD is failing.

1. Is there any restriction on number of records a QVD reads ?

2. I am thinking of genrating 5 QVD's with 10 M records each and then fetch records from these 5 QVD's into dashboard (QVW). Is there any mechanism to dynamically distribute records in number of QVD's ?

3. Will my final Dashboard(QVW) will be able to withhold 50 M + records ?

Please suggest and guide on apporach and performance improvement.

Thanks in advance..

Re: Restrict record load in QVD from table/csv

jonathandienst — Fri, 15 Sep 2017 06:35:11 GMT

1. >>Is there any restriction on number of records a QVD reads ?

No, but there may be restrictions caused by your data source.

2. >>Is there any mechanism to dynamically distribute records in number of QVD's?

No automatic mechanism, but you can write your load/save process so that it partitions the QVD

3. Will my final Dashboard(QVW) will be able to withhold 50 M + records?

Technically no (subject to available RAM), but 50M rows is a lot for a dashboard.

Re: Restrict record load in QVD from table/csv

onkarkulkarni — Fri, 15 Sep 2017 10:26:01 GMT

Thanks a lot Jonathan,

For 50 M records to load in QVD, How much RAM will required ? Curently I have 16 GB on my system, and QVD is failing post 25 M..

I am planning to have a logic which will load records in different QVD's in chunk.

Regards,

Onkar