Incremental load data from DynamoDb

[eicki]

Hi,

First of all am I right that I can load data from an AWS DynamoDB table into QlikSense at QlikCloud as described here: https://help.qlik.com/en-US/cloud-services/Subsystems/ODBC_Connector_help/Content/Connectors_ODBC/Dy... ?

If yes how do I implement an incremental load? My DynamoDB table gets half a million new rows each day and we keep them for 180 days, thus we have 90 million rows, doing a full scan on the Dynamodb table every time I create a report in QlikSense does not sound right and is probably quite expensive.

Björn

[Mark_Leifeld]

HI @eicki, 

if possible it is always a good idea to implement an incremental reload. 

Here are some tings you have to check. 

1. Can you identify a unique row? Keyfield ? 

2. Can you identify new or modified rows ? Fields like "Modified_date" "last_change" or "ETL_Timestamp" ? 

3. Any Upset logic ? Only adding new rows ? Adding and changing ? Add, change and delete ?

Here are a few example. Based on QVD and excel.

Only add: 

Add & change 

Add, change and delete