Increment Load

[its_rajvir]

Dear Experts,

I need your suggestion. I am working on an app in which i need to keep last 30 days in the app. The data is huge so i cannot run the whole data daily. so i want to use increment load to load only the updated records. The data will refresh every 30 minutes. Every time the app run it should show the new data + the data within the range of 30 days. the old data should delete from the app. 

Any suggestions how i can achieve this pleaseeeeee.

Thanks in advance.

[JHuis]

there are many different approaches for this: 

Can you give me an example of dummy data? I preferably need an keyfield, registration date/timestamp etc. 

And does the new dataset also have new and updated records?

[its_rajvir]

Hi @JHuis Thankyou for your Reply.

Below are the main columns in the table:

Customer ID	Equipment ID	Equipment Name	Type of Txn	Txn Value	Txn Date	Txn Time
1231	0d76	abc	A	340	DD/MM/YYYY	HH:MM:SS
3452	06f6	erd	B	267	DD/MM/YYYY	HH:MM:SS

data is refersing every 30 mins.

[JHuis]

Dear:

Can you say what this is for your dataset?

1. Record Timestamp or ID: Ensure that your source data has a column that indicates the timestamp or a unique identifier for each record. This is crucial for identifying new or changed records during subsequent loads.

2. Identify New/Changed Data: During each subsequent load, you need to identify records that have been added or modified since the last load. This can be done using the timestamp or record ID column. 

[Tanish]

Hi Rajvir,

Create a variable 

Let vMinDate =  Date( Today()-30  , 'DD/MM/YYYY' );

Trace '$( vMinDate)';

Load ......

From 

Where [Txn Date] >= $( vMinDate) ; 

This way you will always have last 30 day data only in the Dashboard.

Thanks,

Tanish