Data load

[shephali_23]

I am fairly new to Qliksense and want to understand the data loading.

I loaded a huge table of around ~300 million rows. To avoid reloading, I commented the script and loaded another tables via another script load. But this resulted in the loss of the previously loaded table. Is there a way to restore the old table? The old table load took hours.

[martinpohl]

when you click on reload, all datas are reloaded

What can you do:

once, create a qvd from your datas (load field from datasource; store table into [lib\name\data.qvd] (qvd);  (see here https://help.qlik.com/en-US/sense/November2019/Subsystems/Hub/Content/Sense_Hub/Scripting/ScriptRegu... )

so you don't have to wait that the datas are transfered from the database. qvd load is extremly faster.

Onother speed solution is only to select fields you need, don't use select *

Regards

[shephali_23]

Hello, The table which was loaded doesn't exist at all because i commented that part of the script and then again did a "LOAD DATA" to load other smaller tables. My question here is if there is a way to get the big old table back?

[Kushal_Chawda]

you probably want to use Partial load. When you load first big table then loading other table with adding prefix Add load will add only those tables when you do partial load.

I think in qlik sense there is no partial load option for which you may want to use extension. Please see below link

https://support.qlik.com/articles/000046093

[jonathandienst · Accepted solution]

Short answer: No

A normal reload drops all data and then loads according to the script. To avoid loading for hours every day, you could look at incremental loading and partial reloads, or if the table is static, store it in a QVD and load from there.

Incremental loading involves storing the invariant records of the 300 million rows in a QVD file and only loading new/changed data from the database. If the QVD is loaded without making changes, 300 million rows should only take a couple of minutes, so it is much faster. It is quite an advanced concept with may factors to consider, so you might want to create a table in your source with a smaller subset (say 1m rows) and develop the incremental load on that. There  are many articles and blog posts in the community on incremental loading, so I suggest you start there.

Partial loading is a special case where data is added or replaced without dropping the entire set. You can read up more about that in the help, but it will involve some rethinking and rewriting of your code and you will need to consider how to update the 300 million row table, probably via an incremental load.

A final option to consider is whether you need all 300 rows and all the columns. reducing your data down to what is needed may improve performance of the reload AND the front end as well as reducing RAM requirements in your server.

[shephali_23]

Thank you. This really helps.