Incremental Load Problem!

[#1]

Hi All,

I tried to do the incremental load with the help of the document which is given in the below link.

http://community.qlik.com/media/p/125837.aspx

But as per me it is not working . Requesting you to let me know what changes I need to do so that it works as expected.

Explanation about the problem given below:

~~~~~~~~

I tried the incremental load application with the data of mine. I am facing few issues when I was trying with my data. It is not working as expected. Explained about the problem in the screen shot which is given below. Hope you can look into it and provide some solution for the same.

I have attached the document which I was trying with.

~~~~~~~~

Thanks and Regards,

Rikab Kothari

[#2]

Sunil Jain wrote:
I Implemented Rob's Logic on your document. pls check attached File. If you any query pls reply me.
if you want to update data from specific date then



ELSE


// QVD does not exist

LET INCREMENTAL_EXP = '';
// No QVD. Force full reload
//LET INCREMENTAL_EXP = 'WHERE ModDate >=' & MakeDate(2005,01,01); //?? Could also set it to some floor date


END

IF

 pls MakeDate(2005,01,01) Change Date here. <div></div>

Hi Sunil,

I don't want to do update the data from specific date. I just wanted to do the incremental load whenever I reload the data. Have seen the document which you have attached. Think it is working. If any doubts are there will get back to you.

[#3]

Sunil Jain wrote:
Uncomment following part and Add date from which you want to update record in QVD.
MakeDate(2005,01,01) Change Date here. <div></div>

I don't want to load the data on specific date. I have not tried with your suggestion yet!

I was just wondering what made you to give this as your suggestion to load the data using specific date. Did I asked you anywhere that I want to load the data on specific date. Please let me know!

[suniljain]

Dear Rikab,

Specific Date in a sense that suppose I have 10 yrs data and first five years data is freeze and never require to update so in that case we can give date for updation of data of last five yrs only. there is no others meaning of that statement.

Regards

Sunil Jain.

[#5]

Hi Haneesh,

I am clear with your explanation on why are doing concatenation and why we are loading all the data in to the qvd again.

Many thanks for the response!

[#6]

Hi kerempekcabuk,

Thanks for your response! I don't want to reload the document on hourly or minutely based. I just wanted to do the incremental load whenever I reload the data.

Please reply me when you are free. It's not that necessary to reply me in that lunch time :-). Take enough time for your lunch if time is there please reply me.

[#7]

Haneesh wrote:
Yes, it is loaded from the QVD when ran second time. I ran it for the first time and it is loaded from excel. I ran it again and it is loaded from QVD. Please refer the attached log files. The 'incrementalload.qvw.2010_08_13_13_03_39.log' file is generated for first run and 'incrementalload.qvw.2010_08_13_13_04_02.log for the second. You can find the details of data loaded here.
In this case since the source is excel, there might not be huge difference in reload time. But, when you have DB as source, which is in another machine, then there will be huge time difference between DB load and QVD load. I guess, this clears your doubt. <div></div>

Yes I saw the log files which you have told me to refer. I will surely try to load the data from the database so that I can feel the difference.

[#8]

Hi Jonathan,

Yes you are correct I misunderstood on incremental load concept. These days I was thinking that once the data has been loaded first time and from there on it will load only those data which are new(inserted,modified and deleted).

Many thanks for explaining me on how exactly incremental load happens. It really helped me!

There are some doubts in the script hope you will help me out there also.

[#9]

Sunil Jain wrote:
Rob's Methos is faster because It is work on the basis of indexing on primary key.
It takes lesser time to update and insert record in existing QVD. I tested this logic on 6 crore of records. And Happy with the performance of Rob's Application. <div></div>

Many thanks for sharing your experience on Rob'd method. As suggested I will also follow Rob' method only.

By the way I have one more doubt in the Rob's method. In his document he is mentioned that "It Does Not Handle Deletes" but I want that also to be taken care in my document. Is there is any way to handle the same with this method.

Please do let me know!

[#10]

Sunil Jain wrote:
 Specific Date in a sense that suppose I have 10 yrs data and first five years data is freeze and never require to update so in that case we can give date for updation of data of last five yrs only. there is no others meaning of that statement. <div></div>

Oh yes! That seems to be a better option which I never thought of it. Will defiantly take this in to consideration as this will help to increase the performance.

[suniljain]

Dear Rikab,

What are the probabilities to delete customer in your case ?.

I think It rarely happen in real Business Scenario. Because If Ones Customer Created It affect all the functional data like sales , Finance, Inventory. so I think no one want to disturb their Business process in reality.

Regards

Sunil Jain.