Skip to main content
Announcements
Join us at Qlik Connect for 3 magical days of learning, networking,and inspiration! REGISTER TODAY and save!
cancel
Showing results for 
Search instead for 
Did you mean: 
khalander
Creator II
Creator II

Incremental load by using qvd date

Hi Everyone,

I have one requirement like system has to do incremental process based on date in particular qvd

For Example. I have source file like

IDNameDate
1A01/01/2016
2B31/01/2016
3C01/02/2016
4D24/02/2016
5E01/03/2016
6F15/03/2016

Requirements:

1) Store the month wise data into particular month qvd like jan data into X_012016.qvd, Feb data into X_022016 and so on.

2) Now i need to write one for loop like system has to check the dates in oldest qvd first( in our case Jan month qvd) and then it should do incremental process( Insert or update) from DB and jan qvd should contains updated jan data. and it has to check for all qvd and hs to update data for all month qvds.

Kindly do the needful

17 Replies
khalander
Creator II
Creator II
Author

Hi Rahul,

PFA for sample data.

I am using same data

Vasiliy_Beshentsev
Creator III
Creator III

rahulpawarb
Specialist III
Specialist III

Hello Dada,

Please refer attached script as well as zip file. This will solve the purpose.

Thank you!

Rahul

khalander
Creator II
Creator II
Author

Hi Rahul,

Attachment is missing.. can you attach the files

rahulpawarb
Specialist III
Specialist III

Hello Dada,

I could see the attachments. However, I have re-attached it herewith.

Also, refer below given modified script:

DateListTemp:

LOAD

  Distinct LEFT(RIGHT(Date([Transaction date],'DD/MM/YYYY'),7),2)& RIGHT(Date([Transaction date],'DD/MM/YYYY'),4) AS MonthYear

FROM [Copy of incree load data.xlsx]

(ooxml, embedded labels, table is Sheet1);

NoConcatenate

DateList:

LOAD

  MonthYear

Resident DateListTemp

ORDER BY 1;

Drop Table DateListTemp;

//EXIT SCRIPT;

FOR i=0 TO NoOfRows('DateList')-1

  LET vMonthYear = PEEK('MonthYear', $(i), 'DateList');

  LET vQVDPath = 'QVDs\'; // Set QVD Storage Directory

  Sales:

  LOAD

    Region,

    id,

    product,

    sales,

    [Transaction date]

  FROM [Copy of incree load data.xlsx]

  (ooxml, embedded labels, table is Sheet1)

  Where LEFT(RIGHT(Date([Transaction date],'DD/MM/YYYY'),7),2)& RIGHT(Date([Transaction date],'DD/MM/YYYY'),4) = $(vMonthYear);

  // Check to see if this is the first reload. if it is, skip this step

  if not isnull(QVDCreateTime('$(vQVDPath)Sales_$(vMonthYear).qvd')) THEN

    Concatenate(Sales)

    LOAD *

    FROM $(vQVDPath)Sales_$(vMonthYear).qvd(qvd)

    WHERE Not Exists(id);

  end if

  // If data exists within table, store to QVD.

  if NoOfRows('Sales') > 0 THEN

    STORE Sales INTO $(vQVDPath)Sales_$(vMonthYear).qvd;

    Drop Table Sales;

  end if

Next

Drop Table DateList;

Thank you!

Rahul

khalander
Creator II
Creator II
Author

Hi Rahul,

Script is perfect to my requirement. Only thing is all values are loading from sources while doing incremental load for particular month. i have inserted new record for second load instead of loading only new record but it is loading all values. But it is really helpful and perfect script..

Thanks a lot

khalander
Creator II
Creator II
Author

I am not able to mark it as correct answer.. Can you help me how to do that

arixooo123
Creator III
Creator III

Dear Rahul,

Would you please check my question and let me know your suggestion:

Incremental Loading on monthly Partitioned QVDs