topic Data Load from SQL; QvOdbcConnectorPackage.exe.config Settings in Connectivity & Data Prep

Data Load from SQL; QvOdbcConnectorPackage.exe.config Settings

brandonblaine — Tue, 21 Dec 2021 16:57:48 GMT

Hi,

Currently, when I load data from SQL into a Qlik Sense QVD, the Data Load Progress window shows that about 3,000 rows at a time are loaded into the QVD.

Can this be increased? The QVD is super fast obviously because it is all in memory. Loading the QVD is pretty painful though. Is there an equivalent to the SSIS fast load option?

I browsed the QvOdbcConnectorPackage.exe.config file and nothing stood out except maybe:

Here is the load script. Nothing fancy.

Charges:

Load *;

SQL SELECT * FROM dbo.CHARGES;

Store Charges into [lib://FileSystem/Charges.qvd] qvd;

Drop table Charges;

Re: Data Load from SQL; QvOdbcConnectorPackage.exe.config Settings

felipedl — Mon, 29 Jul 2019 18:38:00 GMT

Weird behaviour,

Are you able to only load all records to memory (removing the store)?

Any error messages appear when storing to the QVD?

Dumb question, are there more than 3000 records on the table?

It should do quite well in loading data from SQL, i've done a few millions quite fast and storing without a problem, can you specify the exact actions are made and what goes wrong with a printscreen?

Re: Data Load from SQL; QvOdbcConnectorPackage.exe.config Settings

brandonblaine — Mon, 29 Jul 2019 18:45:16 GMT

Hi felipedl,

Thanks for the reply. This seems to be the default regardless of if I am loading into a QVD or straight into the app from the SQL source. The table I have is ~2 million rows. It is only one table which is the result of a de-normalized query which I do because I don't believe Qlik handles complex data models very well.

I'd forgo the QVD step if I could but the data manager will not process that much data without it being in a QVD.

Re: Data Load from SQL; QvOdbcConnectorPackage.exe.config Settings

brandonblaine — Mon, 29 Jul 2019 21:39:30 GMT

I just had the IT team increase the number of cores on the VM from 4 to 8. I'll have to check the logs to see if this helps. The VM has 60GB of RAM.