Incremental load approach

[#1]

Hi,

I'm currently building an app loading data from hundreds of Excel files.

One new file each date containing about 10,000 rows. I'm now about to enable incremental load in my script but not sure which approach to use. Is it for instance possible to exclusively load/add "todays file"? Last modified file etc?

I guess using unique ID-keys (I have to put them toghether myself) searching through millions of fields would be about as time consuming as doing a full reload?

How would you solve it?

Thanks in advance,

Olle

[#2]

Hi,

in this case , Incremental load resolves time consuming on load.

this can be done in three stages as per the requirement.

STAGE 1: only load new records into the document

STAGE 2: adding new records along with updated records in the Live data

STAGE 3: load records which are newly added, updated and deleted on Live data

Pls refer this QVW file for three stages.

This will help you .

Regards

Charitha.

!

[Anonymous]

Sure , you can do it.

Go throw the file storage, find the latest one with FileTime function and load to the QVD (Concatenate with previously loaded data)

[#4]

Thanks for your answer Lakshmi.

Looks good! However, I do not have an ID-field in my source data. I need to asseble one myself as per below.

Date & ':' & OriginalTime & ':' & Reference & ':' & SettlementAmount As TransactionKey

This prevents me from executing the IncrLOad in stage 1, since TransactionKey is not an actual field in the source doc.

Any idea how to get around this?

Thanks in advance,

Olle

[#5]

all the concatenated fields are existing in the Live data souce ?

- Charitha

[#6]

All of the fields are existing except a unique ID-field (called OrderID in your example). That is why I need to create one myself. 

-Olle

[#7]

yes , at STAGE 1 we have conditional field called Orderdate only .though it might be combination of all fields you can write the below sample for STAGE1

TableName:

Load *, Date & ':' & OriginalTime & ':' & Reference & ':' & SettlementAmount As TransactionKey

where DateField <= ' <Date>';

[#8]

Thanks for your answer.

However not sure I'm following

The problem occurs  because TransactionKey is not an actual field in the document.

FROM
[IncrementalLoadTest\2014-01-02.xls](ooxml, embedded labels, table is Incoming)
WHERE Date(Date,'MM/DD/YYYY') >= '$(vMax)' and Not Exists(TransactionKey);

[#9]

in my source data document that is!

[#10]

what i am trying to say is

Load *, Date & ':' & OriginalTime & ':' & Reference & ':' & SettlementAmount As TransactionKey

FROM
[IncrementalLoadTest\2014-01-02.xls](ooxml, embedded labels, table is Incoming)
WHERE Date(Date,'MM/DD/YYYY') >= '$(vMax)' and Not Exists(TransactionKey);

this will works rght in you case ?