Skip to main content
Announcements
Qlik Community Office Hours, March 20th. Former Talend Community users, ask your questions live. SIGN UP
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Incremental Load

Hi to all,

                I am just try to do incremental reload....actually i just stored my old data in QVD then am trying to reload my new data and Concatenate to my old QVD into my new data...but my old data and new data everthing will be in same place only and i cant reload using max of date because in my new excel old data will be updated so how can i update my new excel file into the old one..please its urgent...thanks in advance to all

14 Replies
ashfaq_haseeb
Champion III
Champion III

Hi,

You need to look at

Insert, Update and Delete methodology.

Have a look at attached document .

Regards

ASHFAQ

hariprasadqv
Creator III
Creator III

Use of Exists and not exists with where condetion will works fine.

Not applicable
Author

before itself i went through it....but this thing is not will work for me....but that document is helpful...

am looking some other solution..else i want to know how to store a row values in variable using for loop...

shambhub
Creator
Creator

Hi Narayan,

Please can you provide table structure from both the sources(QVD & Excel) and please mention Primary key also.

Are you doing incremental load for 1) Insert ,or 2) Insert Update or 3) Insert Update Delete ?

Regards

Shambhu

Not applicable
Author

Hi,

try this code ..for insert only

Max:

LOAD

     Date

FROM

(qvd);

Res:

Load Date as ModDate

Resident Max order by DATIME Desc;

Let MaxTimeStamp=Peek('ModDate',0);

Let MaxTIMESTAMP=Timestamp($(#MaxTimeStamp),'YYYY-MM-DD-hh.mm.ss.ffffff');

Drop Table Res,Max;

//Select from SQL(Database whatever you using)

QV_Table:

SQL SELECT *

    FROM ABP WHERE DATIME >= '$(MaxTIMESTAMP)';

CONCATENATE

LOAD  

           *

    FROM

(qvd) WHERE NOT EXISTS(COMPOKEY);

STORE QV_Table INTO 

(qvd);

Drop Table QV_Table;

NOT EXISTS(COMPOKEY)  is used for insert and update only.

For insert only don't use this where clause.

Regards

Vimlesh

its_anandrjs

Hi,

Try Insert Incremental load here that help you. For that see the example load script for that and make changes according to your end

Step One

Final:
LOAD
Id,

      UpdateTime,
FROM
Log_Final.qvd(qvd);

let
vMaxTime = Peek('UpdateTime',-1,'Final');
DROP
Table Final;

 

Step Two

Log_Details:
LOAD
Id,
     UpdateTime

FROM
Log_Update.qvd(qvd)

Where UpdateTime > $(vMaxTime);


//Old Data loaded here

Concatenate
LOAD
userId,
     updateTime
FROM
Log_Final.qvd(qvd);

Updated:
NoConcatenate

LOAD
userId,
     updateTime

Resident Log_Details Order By updateTime asc;
DROP
Table Log_Details;
STORE * from Updated into Log_Final.qvd(qvd);


Have a look attached document file for this


Note:- One you have to manually create the Log_Final.qvd first then do incremental load on the new data.


Regards

Anand

Not applicable
Author

Test.PNG

here i attached the snapshot of my data structure....incase i had added one more excel called Transaction-Apr2014.xlsx i need to reload only the Transaction-Apr2014.xlsx...and Concatenate the Qvd file into this Excel file.

Peter_Cammaert
Partner - Champion III
Partner - Champion III

Do you really want to load historical data into every new Excel file that passes by?

Why not the other way around? Create a QVD (or a resident table in a QVW document) that contains the data from the historical QVD as well as the data from the latest xlsx file?

Not applicable
Author

order by DATIME Desc......This DATIME is wat is it function or a field...if it is field means let me know clearly.