Incremental Load issue

[brijesh1991]

Hi,

I had a problem with my incremental load of data.

When I am taking full load of a table, the qvd size is 1 GB, but when next day I am taking the incremental load of the same table, qvd size reduces dramatically to 400 MB.

My incremtnal script is somewhat like this:

------------------

Let

vThisExecTime=chr(39)& date(now() ,'YYYY-MM-DD hh:mm:ss') & chr(39);

Let

vQdvExists=isnull(QvdCreateTime('..\Data\Qvd\demotable.qvd'));

if ($(vQdvExists)=-1) then
let vIncrementalLoad=0;
else
let vIncrementalLoad= ' where "demotable".ADW_LAST_UPDT_DATE >= ' & chr(39) & $(vLastExecTime) &chr(39)& ' and "demotable".ADW_LAST_UPDT_DATE < ' & chr(39) &$(vThisExecTime) & chr(39)& '

ENDIF

if ($(vQdvExists)=-1) then
demotable:
select  * $(vIncrementalLoad);

ELSE

demotable_Temp:
select
* $(vIncrementalLoad);

Concatenate (demotable_Temp)

LOAD * FROM
[..\Data\Qvd\demotable.qvd]
(qvd) where not Exists(Contract_Number);

NoConcatenate
demotable:
LOAD * Resident demotable_Temp;

drop Table demotable_Temp;

ENDIF


store demotable into ..\Data\Qvd\demotable.qvd;

drop Table demotable;

let vLastExecTime=vThisExecTime;

exit Script;


-------------------

Your feedback is too much important to me. . .

BRIJESH

[IAMDV]

Hi Brijesh,

Check your Exists() function. I think you need two parameters in Exists() function. You might be missing the data in Incremental Load. Debug your Incremental Load script and see the paramters & where clause.

Good luck!

Cheers,

DV

www.QlikShare.com

[brijesh1991]

Thanks for your time, Deepak,

My script is uptodate, it's correctly debuged without error and I have checked exists function which is also ok(since contract_number is primary key, it checks whether incremented data has same contract again, then reject, else concate). . .

Actually My original script is modified for confidential reasons, otherwise where clause is also ok, i have double checked.

I don't know why size is reduces. actually size should be greater than yesterday's data. . .

Let me know if you find something!

[IAMDV]

Hi Brijesh,

Please can you test removing Where Exists() and see if you have full data? If you see full data then it means the Exists() function is not working as expected. If the problem still persist then we need to look further. Also, why do you need Resident Load? Instead can't you rename the Table Name? It would save the load time...

NoConcatenate
demotable:
LOAD * Resident demotable_Temp;

drop Table demotable_Temp;

ENDIF


store demotable into ..\Data\Qvd\demotable.qvd;

drop Table demotable;

let vLastExecTime=vThisExecTime;

exit Script;

Good luck!

Cheers,

DV

www.QlikShare.com