topic Re: Incremental only update.. in App Development

Incremental only update..

Tool_Tip — Thu, 12 Sep 2024 07:53:24 GMT

Dear all,

I have table such as below without any modification date fields and every day the period is changes based on backend calculations.

Day:1

ID Period

1 2Yrs-3months-10days

2 4Yrs-5Months-0days

3 0Yrs-1Month-10days

Day:2

ID Period

1 2Yrs-3months-10days

2 7Yrs-2Months-0days

3 0Yrs-1Month-11days

Our requirement is display the latest Period for each ID. How can we achieve this.

Re: Incremental only update..

marksouzacosta — Thu, 12 Sep 2024 13:11:15 GMT

Hi @Tool_Tip,

A few questions:

How many records do you have in your data source?
Do you need to keep historical chances?
Can a Period be reduced instead of increased?

Regards,

Mark Costa

Re: Incremental only update..

Tool_Tip — Mon, 16 Sep 2024 04:04:03 GMT

Hi Mark,

Below answeres:

How many records do you have in your data source? Almost 300000
Do you need to keep historical chances? No need, only latest change to display
Can a Period be reduced instead of increased? Either..

Re: Incremental only update..

marksouzacosta — Tue, 17 Sep 2024 12:49:51 GMT

Hmm maybe in your scenario it is better to just do a full load every time instead of an incremental load. How much time to do a full load?

Re: Incremental only update..

Tool_Tip — Wed, 18 Sep 2024 03:52:27 GMT

We have aborted after keep it loading on 24hrs... 😉 so not sure how much total time it will take to completed load.

Re: Incremental only update..

marksouzacosta — Wed, 18 Sep 2024 12:56:24 GMT

Awesome. I would need more details to be able to help you. Like, what is the data source, what are the columns, sample values (if possible), your current code (if possible). Qlik offer many possible solutions for ETLs. I believe a mix of Data Partition, Incremental Load and some progressive saving QVDs may be a solution for you.

Re: Incremental only update..

ali_hijazi — Mon, 23 Sep 2024 05:45:26 GMT

Hello
You are saying you got 300,000 records
that's a sip of water for Qlik so no need for incremental reload.
the historical data / changes you are talking about are in the data source or in your qvd file?
we need more details about connetion, script, ... in order to resolve the issue

Re: Incremental only update..

Tool_Tip — Tue, 24 Sep 2024 04:16:49 GMT

Its 3000000. My bad.

Obviously data changes in source.

Re: Incremental only update..

ali_hijazi — Tue, 24 Sep 2024 06:29:21 GMT

still 3M is handled easily by Qlik
and here too there is no need for incremental reload
so the question is are you experiencing low performance during loading the data or on the user interface?
if you are experiencing low performance on the user interface then you need to review the schema, otherwise it is the database that is slow in responding; in this case sometmes we change the connector that we are using to connect to the database;
for example we had an oracle database to connect to, and we found out that the OLEDB connection is faster than using the Oracle connector available in Qlik:

last thing your SQL statement is it a simple select * from table or you are making join and other transformations against the data source?

Re: Incremental only update..

marcus_sommer — Wed, 25 Sep 2024 12:53:12 GMT

Like already hinted it's not a big data-set if it's directly loaded from a table. Therefore I assume it's the result of a (complex) query against a much bigger data-set within the data-base - and here is the data-base itself the bottleneck respectively the belonging query (probably with a lot of sub-queries which might be replaced with tables/views and/or may extended to appropriate flags which record is the first/last one).

You may also considering to transfer the complete logic into Qlik by loading just the raw-data from the data-base incrementally and applying the needed transformations on top of it in n incremental layers.