Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Qlik Connect 2026! Turn data into bold moves, April 13 -15: Learn More!
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

SQL table taking too long to store into QVD


Hi,

I have two huge sql tables with Table1 has 4213039 rows

and Table2 has 12710440 rows

when I am loading the tables into QVDs by using

Store Table1 into C:\qlikviewpath\Table1.qvd;

Store Table2 into C:\qlikviewpath\Table2.qvd;

to load the scrit execution is taking more than 14 minutes. could anyone let me know is there anyway to load data into QVD very fast.

Thanks.

27 Replies
Not applicable
Author

Hi please see below

Transactions:

LOAD TransactionID,

    TransID,

   ActualAccountID,

   PayeeAccountID,

    AccountID,

    ID,

    Type,

   TransTypeDomain,

    Category,

   TransDate,

    TransactionAmount,

    Amount,

   PendingAmount,

    AvailableAmt,

    BACSStatus,

    CMSStatus,

   PostedFlag,

   RecFlag,

   VoidTransactionFlag,

   FailReason,

   Reference,

    BatchID,

   PaymentPlanID,

   BDDTransID,

    ReturnedChequeNumber,

    InvoiceNumber,

    InvoiceDate,

    Refund,

    RecStatementID,

    CMSTransID,

    CreditorPaymentTransID,

    FeePaymentFlag,

   CreatedOn,

    CreatedByID,

    CreatedBy,

    ChequeRunID,

    ChequeRegisterID,

    Number,

   ChequeAmount,

ChequeRegisterCreatedon,

   ChequeName,

   ChequeCreditorName,

    ChequeStatus;

SQL SELECT *

FROM "XX".schema."TRANSACTIONS";

flipside
Partner - Specialist II
Partner - Specialist II

You have two issues here, then.

Firstly, regarding the SQL query, how does it compare in time to execute this part in comparison to loading the same query on your SQL server? The lack of indexes may be part of the issue, are you using any joins or WHERE clauses? And how does your datamodel look after loading, are there any synthetic keys being created?

Secondly, I'm assuming the QVD is writing to a local drive and not a networked or shared drive?  The time for this is probably just down to the speed that the server can write the file - are all the fields really necessary? Without upgrading the server you are probably left with trying to reduce the file size or change your QVD strategy.

flipside

greg-anderson
Creator II
Creator II

The query looks pretty straightforward, assuming 'TRANSACTIONS' is a table rather than a view.  The lack of indexes might slow it down, but with a straightforward SELECT * and no WHERE clause, that should not be a major problem.

I see that flipside just posted what I was about to say.  Without knowing more, I'm guessing the issue is the network speed between you and the database server and/or the network speed between you and the file destination.

I would look into splitting the table into at least 2 QVDs, if possible.  Link them by Transaction ID.

Hope this helps.

-Greg

Not applicable
Author

Transactions table is taking 5 mins 10 seconds in sql server and there are no indexes in this table and Table2 is taking 6 mins 11 seconds in sql server and there is one clustered index and two non clustered indexes in Table2.

Secondly I am testing the application in D drive but in different folder and as Live files are in different folder in the same drive.

So, in this how could I proceed please any help.

Thanks.

jagan
Partner - Champion III
Partner - Champion III

Hi,

Instead of select *, specify the columns you are using in dashboard in Select query, it reduces the data to load and also it reduces the QVD size also.

Regards,

Jagan.

flipside
Partner - Specialist II
Partner - Specialist II

Are they the times when run as queries within SQL Server? If this is the time to load data into Qlikview then that's less than 3 minutes to store data into QVDs or am I misunderstanding the facts?

greg-anderson
Creator II
Creator II

When you post the times, they don't really look bad to me.  SQL Server is taking most of the time, whether in network lag, server processing, or waiting in a queue for processing (if it's a busy server).

rbecher
MVP
MVP

Fields with high cardinality like IDs are blowing the in-memory table and the QVD. You should really think about which fields you can omit here.

- Ralf

Data & AI Engineer at Orionbelt.ai - a GenAI Semantic Layer Venture, Inventor of Astrato Engine
rajeshvaswani77
Specialist III
Specialist III

Hi Amelia,

Do you see optimized load when its loading?

Also, on your development the speed could be less, on server will be much lesser.

To add more, this will be first time. At later time you will do incremental load and take only few records that have changed and concatenate to the QVD.

Guess this should be in seconds.

thanks,

Rajesh Vaswani

Not applicable
Author

Hi,

Could you tell me how to see Optimize Load and incremental load.

Thanks.