Uploading 1.78 billion records from 1 Table

[#1]

Hi All,

I need to update 1.78 billion records from one table and needs to store in the qvd files.

The table has only 8 columns.

Which would be the best way to load it?

1) I tried loading directly, loading never even started.

2) I tried loading by spliting into 4 columns, it started loading and it took 15 hours to load 53 million records and hung.

3) I tried adding indexing column but it is taking more time for indexing itself and it never started loading.

My system RAM is 4GB and I can see Qlikview alone consumes 2GB of RAM when any of the above tasks is running.

Pls. provide an optimized and possible way to update such a huge data.

I am ok to have multiple QVDs as long as I can load them into Qlikview after storing into multiple QVDs.

Thanks.

[Or]

First, you need to make sure your database can handle the load. This isn't QlikView related (e.g. indexing) - it's something you have to take up with your DBA.

Second, you really shouldn't be running a load of some two billion records on a machine with 4GB RAM. You'll wait forever and a half for your machine to finish all the required swapping. A much stronger machine would be needed to get this done in a reasonable span of time (not to mention deal with the eventual result).

[#3]

Thanks orsh_,

Is it good to load the entire 1.78 billiion records by spliting it into multiple segments.

Like, loading first 25% in 1 qvd and then next 25% in next qvd and so on.

or

Creating 1st temp. tables with first 25% of records and 2nd temp tables with second 25% of records and so on?

Then, i will load all temp tables into different qvd files.

Is this a good approach? any drawbacks?

[marcus_sommer]

I don't think you will ever happy with this amount of data and your machine ... even all data are stored in qvd you will only be able to handle each time a few piece of it ...

To create the qvd split the data in pieces of about 1 millions and after a certain number of loop-runs disconnect the connection, clean the RAM (also from database-server) and so on ... perhaps it required restarts ... and waiting ...

- Marcus

[peschu123]

I know this is off topic...BUT ...

there are mobile phones with that amount of RAM on the market... No offense, it just crossed my mind by reading your post.

[#6]

Thanks Marcus and Peschu for your reply.

Based on your replies.. I can understand that it is not possible to load 2 billion records in Qlikview with a 4GB RAM in any way.....

This is the first time, I am loading such a huge data, so I just want to know if there is way or not..