Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Incremental reload

Hi,

I have a QlikView application which loads the data from a
oracle database...

SELECT * FROM DB."DATE_DIMENSION";

Is there a way to do an incremental update of the data (only reload new data)
without loading the entire data of the db.

How is it possible...?

62 Replies
shumailh
Creator III
Creator III

I wish no one would get into this flat file crazy format, the development for ETL on this crazy data will take lots of time…
Ok Dude NP I am telling you this step by step. Actually Incremental load is based on the requirement i.e. source and data which you are getting.

We need to take care of three areas Input (Source TXT/ FLAT FILE), Process (Transformation, ETL, Database i.e. QVD) and Output (QVW).

INPUT:

First I would like to explain you about the source, the source data can be in any format and is not delimited but the files in daily folders has the equal format and incremental/appended. So the source where I am getting data is not a database structure it's actually a flat file which can be present in some network folder on daily basis. Eg: for July 2010 data I have following C Drive folders

C:\Source\20100701
C:\Source\20100702
C:\Source\20100703
C:\Source\20100704
C:\Source\20100705
C:\Source\20100706
.
.
.
.
.
C:\Source\20100730
C:\Source\20100731


Consider we have 2 files in each daily folder, one is incremental and other is appended, and take them as file A and B respectively.


PROCESS:


Create a new QVW file to generate QVD for flat file A and separately for flat file B. In each qvw file you need to develop the transformation mechanism for the file into structure / your required database format and store it into NEW QVD, make sure your coding should be that much flexible that on daily basis you can schedule it into publisher. So you have a separate code for incremental and appended data. Then when you schedule it in publisher this qvd will automatically be updated with the new data and other will be updated with the appended data.

In my prior post I mention about the configuration file this is the file where you can define the source path, file format or you can define some batch file if the flat file in a zip format to extract.

After development you need to schedule the qvw's file in publisher as per the availability of source file.

OUTPUT:

This is your final file where you can see and perform all the analysis on daily basis and will be scheduled in publisher as well. For output you need to develop and design a qvw file which is getting data from QVD file which you generated in process level.

Shumail

Not applicable
Author

Hi Shumail,

Thank you so much for your effort! But sorry to tell you again that I am not able to understand and implement your ideas practically.

Still I am thinking why you are not been able to work in my document. I feel that it will be easy for me to understand how incremental load works. Kindly think about working on in my document so that is easily understandable.

Not applicable
Author

Hi Rob,

I referred your cookbook for doing the incremental load. It is really an wonderful work and it will be helpful for everyone. After that going through the script of the incremental load found that it does not handle the deletes.

I wanted to perform delete also! Do you have any other incremental load document which can handles delete also(Including insert and update)? If not can you please create the same and update the cookbook. It will be very useful for all those who wants to handle deletes in incremental load.

Let me know one it is updated!

Not applicable
Author

Hi John, This code is very similiar to my document "Inceremental Load"in Shared Qlikviews "How To" part.

Not applicable
Author

thanks man it really good.works fine

Not applicable
Author

Hello Attitude,

I have a similar requirement like mentioned by you here. I have 6 QVDs from a Oracle DB, I was wondering when I do the incremental load will it update the daily changes only or reload the whole QVD and then update?? Can you please help me with the same issue I am struck with like you had sometime back.

Thanks in Advance,

ANDY

suniljain
Master
Master

The most difficult case to handle is when records are actually deleted from the source database between script executions. The following conditions apply:

· The data source can be any database.

· QlikView loads records inserted into the database or updated in the database after the last script execution.

· QlikView removes records deleted from the database after the last script execution.

· A field ModificationDate (or similar) is required for QlikView to recognize which records are new.

· A primary key field is required for QlikView to sort out updated records from the QVD file.

· This solution will force the reading of the QVD file to "standard mode" (rather than "super-fast mode"), which is still considerably faster than loading the entire database.

Script example:

Let ThisExecTime = Now( );

QV_Table:

SQL SELECT PrimaryKey, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#

AND ModificationTime < #$(ThisExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVD

WHERE NOT EXISTS(PrimaryKey);

Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;

If ScriptErrorCount = 0 then

STORE QV_Table INTO File.QVD;

Let LastExecTime = ThisExecTime;

End If

Not applicable
Author

Hello Sunil,

Thank you for the quick reply, but the problem is I have a Primary field called TIME_PERIOD_ID from where I have to get the 2 variables? the format of this from Oracle DB is somethinglike below, where 2011 is year, 03 is month, 1 is normal week and 13 is Week number. So I want to try Incremental load today and tommorow and check if its working as expected. According to what I think I will get all the sales data for this date and tommorow or next week, when I rerun the script with variables, then it should look for Week 14 data only and load and not the whole data right? Also how to use my Time period Id for the below script??

WHERE



ModificationTime >= #$(LastExecTime)#

AND



ModificationTime < #$(ThisExecTime)#;



,201103113' ,

Thanks a TON,

Not applicable
Author

Hi Neetu Singh,

I just gone through the attached rar file but i need small clarification on reloading part.If Qvd exists the condition goes to QVD and fetch the max date from the date field and does the next reload based on that date,other wise it take full reload rt?Here my concern is,is it possible to fetch last six months data instead of full reload ?If aware,please ping answer on this query.

Thanks in Advance

Not applicable
Author

I am having the same issue the On incremental load there has no primary key being there in table due to which there is being a problem of doing incremental load. Please help in this matter..