millions of records from oracle server

[manoranjan_d]

Hi i m fetching two tables from the oracle server and making joins with these two tables which consists of  million of records, but i want to reduce the running time, since it taking 3 hrs to load. pls let me know how i can reduce the time to below half hour.

[Anonymous]

Hello Manoranjan,

The load time of the tables can be very dependent on the specs of your PC, however,

there are a few ways to reduce the load time of the tables, which you can control.

1. Make the tables QVDs in a separate application, then load and link them together in the application you want to show

2. Filter the data you are loading in the script by using a where statement.  For instance, load the past month, quarter, or year instead of the entire table

[prabhu0505]

First read no other go you have to pay the price.. After pulling it for the first time store in QVD, it will make your life easy for consecutive loads.

Think about implementing "Incremental load" so you need access the source only for delta.

[Clever_Anjos]

You must retrieve all records everytime?

Please check your manual (or community) about using QVD files, so "old" records could be retrieve very fast

In other hand, why don´t you join your tables using a regular SQL statement before loading into QV?

[maxgro]

incremental load, if possible

and some more info will be useful

how long does it takes to read from oracle db?

how long to process (join, etc, ....)?

qlik script and qvw log

[Anonymous]

Hi,

1) Take only required Columns or fields from Database

2) Fetch data without using any filters or functions and create QVDs

3) Use Incremental Load

Hope it will helpful!!

[manoranjan_d]

its taking 1.5 hrs

[manoranjan_d]

ok i will store the millions of records in qvd and  i will use the incrmental load .... but if 200 records added in oracle server and i will use the last date updated function in qlikview , so these will search millions of record and take the 200 records and add them in pervious qvd. so reading of millions of records also take more time..

please clarify.

[maxgro]

incremental:

search million, return (network transfer from oracle to qlik) few hundreds --> ???

+

incremental processing (merge of previously saved qvd and few hundred records from oracle) --> ???

non incremental:

search million, return million --> 1.5 hours

try in a qlik doc and compare the time of the 2 options

other option as already suggested (Clever), join in the db