Increamental load Issue

[aniketsr]

Hi,

I have a table of 73L Record which takes around 1Hr to reload i.e data extraction from Database to QVD.

I Tried loading the data using increamental load wherein i used a date Column which is the transaction date.

But as we use the transaction date in the where clause the Reload becomes more slow and for 20 Mins there is not data fetched.

kindly suggest with any other steps or ways in which the above requirement can be met or else guide with any step if iam missing in the incremental load.

Thanks

[marcus_malinow]

Hi Aniket,

it sounds very much like you need to create an index in your database on your date column. That should speed things up fairly dramatically.

Marcus

[Anonymous]

Aniket

When you say 73L is the L Lakh, and how many in millions is 73L ?

You also say ..we use the transaction date in the where clause the Reload.  Is this where in the SQL database statement or in a QlikView Load Statement ?  Pasting in your load script would help us understand.

Best Regards,    Bill Markham

[Bill_Britt]

Hi Aniket,

This is a little old, but I think it will still apply.

Bill

[#5]

Hi,

You can try this.

Hope it will help you.

Thanks.

[sunilkumarqv]

Hi Aniket,

Please follow steps like below  and see attached .

we can do in  various senarios , Typically below on of that .

//**************************Incremental Load**********************//

//Initial Reload & generate Qvd

ProdMaster:

LOAD ProductID,

    CategoryName,

    ProductName,

    LastUpdateDate

FROM

Sample_Data.xls

(biff, embedded labels, table is [Sample_Data$]);

Store ProdMaster into [..\Data\Qvds\ProdMaster.Qvd];

DROP Table ProdMaster;

//exit script;

Max_LastUpdateDate:

LOAD

    Max(ProductID) As Max_ProductID

FROM

[..\Data\Qvds\ProdMaster.Qvd]

(qvd);

//exit script;

Let  vMaxProductID = peek('Max_ProductID',0,'Max_LastUpdateDate');

Drop Table Max_LastUpdateDate;

Complete_Data:

LOAD ProductID,

    CategoryName,

    ProductName,

    LastUpdateDate

FROM

[..\Data\Qvds\ProdMaster.Qvd]

(qvd);

Concatenate

LOAD ProductID,

    CategoryName,

    ProductName,

    LastUpdateDate

FROM

Sample_Data.xls

(biff, embedded labels, table is [Sample_Data$])

where ProductID>'$(vMaxProductID)';

Store Complete_Data into [..\Data\Qvds\ProdMaster.Qvd];