topic Re: load table 760 million rows in QlikView

load table 760 million rows

felcar2013 — Mon, 26 Jan 2026 18:19:17 GMT

i have a table 780 Million rows and 5 columns, with the following Information

when loaded isolated it takes only 16 minutes, but stucks after it. In case incremental load is not possible, what are other loading Options?

i use the deployment Framework and the Problem i have when connecting directly to the database (extract) and then when loading from QVD. I Need all the data, as per requirement and there is at the Moment no way to reduce it.

thanks

felipe

Re: load table 760 million rows

Peter_Cammaert — Fri, 29 Apr 2016 10:15:52 GMT

When you get stuck loading, does the QlikView application/server become unresponsiv,e or do you get an error message?

Does your QlikView platform have enough memory to handle this amount of data?

Peter

Re: load table 760 million rows

trdandamudi — Fri, 29 Apr 2016 10:28:49 GMT

Based on my past experience, I think it is because of the memory. How much memory do you have ?

Re: load table 760 million rows

felcar2013 — Fri, 29 Apr 2016 11:36:25 GMT

there is no error. see below. after loading appears "x lines fetched" and then it stucks, for over 20 hours

Re: load table 760 million rows

felcar2013 — Fri, 29 Apr 2016 11:40:41 GMT

164 GB RAM

when i load it from the QMC it uses 90% of the Memory, while when doing this manually it takes only 9% of Memory

Re: load table 760 million rows

marcus_sommer — Fri, 29 Apr 2016 12:44:00 GMT

The RAM consumption isn't quit clear with this informations. I suggest you make a restart from the qlikview server services, starts this reload again and monitor the RAM consumption carefully within the taskmanager and you should also pay attention for other activities on this server then not only the qvb.exe will require RAM as well as the qvs.exe and perhaps some other services, too.

- Marcus

Re: load table 760 million rows

felcar2013 — Fri, 29 Apr 2016 16:16:31 GMT

i restartet the Services and realized that the applicaton loads the table, the Memory usage goes slowly up, until it reaches over 90%.

the issue is that this Problem occurs since there was a Migration on the database Server, where the table Comes from. Before QV loaded this table with no Problems.

Re: load table 760 million rows

marcus_sommer — Fri, 29 Apr 2016 17:50:37 GMT

The stucking is clear - there is not enough RAM available. But it looked a bit strange for me that a qvd with 780 M records and only 5 columns takes so much RAM - are you sure that's only 5 columns and you don't performed a wildcard-load on more columns? What kind of data contain those columns?

- Marcus

Re: load table 760 million rows

felcar2013 — Mon, 02 May 2016 10:51:47 GMT

the table contains a Primary key, and the other columns contain no more than 1% of distinct values from total Primary key

i loaded this table isolated directly from SQL and then in the tranform file. in both cases after loading all the data rows, it stucks, where QV Shows the " ..... lines fetched". In the Transform file, the "date" field is transformed with a "floor". The table is loaded really fast, less than 15 minutes (in extract and transform file), but then it stucks and keeps with "... lines fetched". Strange for me is that before our Migration took place, this was not the case. User rights where changed / migrated, Publisher updated and then came the Problem. Is there any place where we can check? (Server side / qv)?

Re: load table 760 million rows

marcus_sommer — Mon, 02 May 2016 11:12:09 GMT

Are there other loadings within these application? If yes there could be occur synthetic keys and/or circular references between the tables and for them could qv need a long time to calculate (by 780 M records many hours and very probably will it need more RAM then available). You could check this within the debugger or with a FIRST statement to limit the data - maybe on 100 records and you will see what happens within the table-viewer.

Another thought is to leave the primary key out of these loadings/tables then are there really reasons for them if you didn't apply an incremental loading?

- Marcus

Re: load table 760 million rows

Marcio_Campestrini — Mon, 02 May 2016 11:18:32 GMT

Felipe

Try to split the TN_DATUM field into two different fields (date and time, if you really need both).

I don't know the period you've to load, but I like the idea of creating multiple QVDs splitting year/month data and after this load to the QVW. With this approach you can look for same load problems.

Re: load table 760 million rows

felcar2013 — Mon, 02 May 2016 11:19:18 GMT

hi,

there are no other loadings, and after loading the table, the application stores it and then Drops the table. there is no possibility to build a synthetic key.

i Need the Primary key to identify which customers participated in a campaign.

Probably the solution is to use incremental load at the end. I just wanted to exactly know, why after the Migration, this does not work

Re: load table 760 million rows

felcar2013 — Mon, 02 May 2016 11:20:47 GMT

thanks, yes, i load the data as floor, not as timestamp, but will split the big table into different tables by year / month and store them as different qvds as you mention.

Re: load table 760 million rows

marcus_sommer — Mon, 02 May 2016 11:32:10 GMT

It will be quite difficult to say what could be a reason without knowing what has changed by the migration - qv release, OS version, storage-environment, virtualization yes/no/changed, other things?

- Marcus

Re: load table 760 million rows

Marcio_Campestrini — Tue, 17 May 2016 12:59:28 GMT

Hi Felipe

Is your problem solved? If yes, please mark helpful/correct questions to close the thread.

Thanks.

Re: load table 760 million rows

felcar2013 — Tue, 17 May 2016 13:15:04 GMT

Hi Marcio,

the solution was to reduce the table to 270 Million rows, deleting old history

for new data we will use incremental load, the source table needed to be updated to allow us recognize new and modified records

regards,

Felipe

Re: load table 760 million rows

Marcio_Campestrini — Mon, 31 Oct 2016 18:54:53 GMT

Have you solved your problem? If you did, please choose the correct answer and help us to keep community focused.