Incremental load data from DynamoDb

eicki — Tue, 12 Nov 2024 20:11:22 GMT

Hi,

First of all am I right that I can load data from an AWS DynamoDB table into QlikSense at QlikCloud as described here: https://help.qlik.com/en-US/cloud-services/Subsystems/ODBC_Connector_help/Content/Connectors_ODBC/DynamoDB/Load-DynamoDB-data.htm ?

If yes how do I implement an incremental load? My DynamoDB table gets half a million new rows each day and we keep them for 180 days, thus we have 90 million rows, doing a full scan on the Dynamodb table every time I create a report in QlikSense does not sound right and is probably quite expensive.

Björn

Re: Incremental load data from DynamoDb

Mark_Leifeld — Wed, 13 Nov 2024 09:49:07 GMT

HI @eicki,

if possible it is always a good idea to implement an incremental reload.

Here are some tings you have to check.

1. Can you identify a unique row? Keyfield ?

2. Can you identify new or modified rows ? Fields like "Modified_date" "last_change" or "ETL_Timestamp" ?

3. Any Upset logic ? Only adding new rows ? Adding and changing ? Add, change and delete ?

Here are a few example. Based on QVD and excel.

Only add:

Add & change

Add, change and delete

topic Re: Incremental load data from DynamoDb in Connectivity & Data Prep

Incremental load data from DynamoDb

Re: Incremental load data from DynamoDb