Skip to main content
Announcements
Qlik Connect 2024! Seize endless possibilities! LEARN MORE
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Incremental reload

Hi,

I have a QlikView application which loads the data from a
oracle database...

SELECT * FROM DB."DATE_DIMENSION";

Is there a way to do an incremental update of the data (only reload new data)
without loading the entire data of the db.

How is it possible...?

62 Replies
Not applicable
Author

Leave everything apart! I want to do the incremental load in the application attached by me.

I don't want to schedule or load it periodically. What I want is I want to load newly inserted,updated and deleted data incrementally instead of loading all the data every time. This is what meant as incremental load i suppose. Correct me if I am wrong!

shumailh
Creator III
Creator III

Incremental load is useful where we have billions of records with large number of fields and we dont want to load the full data every time then we use qvd for incremental load, but here also we need to execute the script for incremental data.

As far as my knowledge is concern automatic loading without execution can not be possible. You are looking for the alernate of publisher cababilities. If you want an automatic load, then you need to schedule it periodically which can only be happen by publisher.

Shumail

Not applicable
Author

Hi Shumail,

I am currently looking for automatic reload only. That isn't possible with the QlikView Desktop? I was not knowing that.

But anyway if it is happening by scheduling it then that is good enough for me so that I don't need to load all the data every time. What I want is to load the only the new(inserted/updated/deleted) data.

So when with out scheduling when I click on the reload button it should load only the new data instead of loading all the data.

Not applicable
Author

Hi Shumail,

Can you please help me out to do the incremental load manually(whenever I reload the data). I need your help very badly!

shumailh
Creator III
Creator III

dude please go up and refer my post on Tue, Jul 27 2010 11:20 AM, i hope that would help u. Sorry i couldn't reply u as i was stuck in some project.

Shumail

Not applicable
Author

Hi Shumail,

Sorry for the trouble. I went up and saw the same with the above dates but didn't find any post of yours with same date and time. Kindly put the same post again now.

shumailh
Creator III
Creator III

Rikab,

There are different ways of doing Incremental load, asaan tareeqa hai kai.... humain ek loop lagana hota hai magar iss main ek dynamic variable hota hai (for each File in filelist ('$(path)CHT*.dat')), check out some of my simple example:

if required tu.... pehlai ek configuration file hum bana saktai hain.... and define some variable here if you need it i.e. configuration.txt


-------Configuration.txt-----------
let CurrentDay= day(now()-1) ;
let CurrentMonth= 06;
let CurrentYear= 2010 ;
let mn='06Jun10';
let prepath='C:\QlikView';

next step is u need to create a qvw file for incremental load / generate incremental qvd.


$(include=Configuration.txt);
let
path = '$(prepath)' & '\Input\' & '$(mn)' & '\Account\' ;
let Outputpath= '$(prepath)' & '\Output\' & '$(mn)' & '\Account\QVDs\' ;

for each File in filelist ('$(path)CHT*.dat')
let
CurrentFileName = right('$(File)', (len('$(File)') - len('$(path)')) ) ;
let LoadDate = mid('$(CurrentFileName)', 7,2) ;
load
'$(File)' as FileName,
FileTime( '$(File)' ) as FileTime,
FileSize( '$(File)' ) as Size
autogenerate (1);

Txn:
LOAD
@1:3 as TxnCode,
@4:22 as Acc_no,
@46:51 as Purchase_Date,
@52:63 as Destination_Amount,
@64:66 as Destination_Currency_Code,
@67:106 as Source_Amount,
$(LoadDate) as LoadDate
FROM [$(path)$(CurrentFileName)] (ansi, fix, no labels, header is 27, record is line);
Next File ;

store Txn into $(Outputpath)Txn_$(mn).qvd ;
//store CH$(CurrentFileName) into $(Outputpath)CH$(CurrentFileName).qvd ;

drop table Txn;


path variable use to take input file and outputpath is using for output qvd file. iss main hota ya hai kai daily basis pai transaction ki ek new file aa jati hai jisai hum TXN_mn.qvd main daal daitai hain simple. try the above code, i hope it would help u
Shumail
Not applicable
Author

Hi Shumail,

I am not able to understand how it works so I my not be able to implement the same thing in my document also. If time permits please try to work in my document or else its ok I will try to find some other solution as I can't expect some one to work on my file if they are not free.

Many thanks for your help these days!

shumailh
Creator III
Creator III

Hi Rikab,

I couldn't reply u since last 2 days was bit busy in month end number, yesterday i wrote a full scenario for u but i couldnt post it as well. In my last post I sent u some basics of my incremental load i think which was bit complex to understand. I think the help given on this is very easy to understand, please see the help and search

Using QVD files for incremental load

I hope this search will help u to understand the scenario of incremental load.Although regarding work on ur file, I think ur file is so simple and there is no need to develop incremental load process, u can either live with the publisher schedule only.

Shumail



Not applicable
Author

Hi Shumail,

Thanks for your response! Yes it is very true that example given in the help is much easier when compared to the one posted by you.

It looks easy when I see that but I am not able to do it practically. So requesting you to apply the same logic as given in the help. No matters if it is very simple. I just wanted to learn how incremental load can be done. So can you please work on it so that I can apply the same logic for the one which is larger.

Please do reply!