topic Re: How to reload latest QVD file only after click 'Load Data' button in App Development

How to reload latest QVD file only after click 'Load Data' button

mudanhong — Sat, 16 Nov 2024 17:31:58 GMT

Below is my incremental loading script:

Test:

Load

where primary key>'xxx';

select * FROM CSV (header on, delimiter ",", quote """") "abc"

WITH CONNECTION (Url "$(vUrl1)");

concatenate load

from abc.qvd

where not exists(primary key);

This only works for avoiding duplicate calculating for the whole abc.qvd each now I start this app task.

But if there is any way that I can only load latest qvd into app RAM rather than loading the whole year QVD file data into app RAM again? I means if there is a way that app could keep the previous run qvd data in RAM and only load required latest data onto the old data in app RAM?

Test:

Load

from abc_2021.02.02.qvd;//2021.02.02 should be defined as latest date

Re: How to reload latest QVD file only after click 'Load Data' button

micheledenardi — Wed, 03 Feb 2021 10:47:35 GMT

How 2021.02.02 should be determinated? is it the yesterday date? Last reload date?

Re: How to reload latest QVD file only after click 'Load Data' button

mudanhong — Thu, 04 Feb 2021 02:42:19 GMT

It is just a specified date. yesterday/today/last reload date whose data has not been loaded into abc.qvd file.

Re: How to reload latest QVD file only after click 'Load Data' button

micheledenardi — Thu, 04 Feb 2021 10:33:28 GMT

In any case you've to load the entire dataset but you can use Binary statement to load data from your datamodel without load data from Qvd/csv and refresh only some tables.

Something like This:

Binary Test.qvf; SET ThousandSep='.'; SET DecimalSep=','; SET MoneyThousandSep='.'; SET MoneyDecimalSep=','; SET MoneyFormat='#.##0,00 €;-#.##0,00 €'; SET TimeFormat='hh:mm:ss'; SET DateFormat='DD/MM/YYYY'; ... ... ... ... SET NumericalAbbreviation='3:k;6:M;9:G;12:T;15:P;18:E;21:Z;24:Y;-3:m;-6:μ;-9:n;-12:p;-15:f;-18:a;-21:z;-24:y'; concatenate(Test) load * from abc_*.qvd where not exists(primary key);

Re: How to reload latest QVD file only after click 'Load Data' button

mudanhong — Fri, 05 Feb 2021 02:57:12 GMT

Hi Michele,

I have tried your way. But I got below error, should I give the absolute path?

Started loading data

The following error occurred:

No qualified path for file: ***

The error occurred here:

Binary QvdIncrementalTest.qvf

Re: How to reload latest QVD file only after click 'Load Data' button

mudanhong — Fri, 05 Feb 2021 03:13:17 GMT

Thanks Michele, I have tried successfully with absolute path of another app.

Re: How to reload latest QVD file only after click 'Load Data' button

mudanhong — Fri, 05 Feb 2021 03:32:01 GMT

But it seems not quicken the app speed😂. I would try a simple app to test its effect on app performance. Thank you for your help.

Re: How to reload latest QVD file only after click 'Load Data' button

micheledenardi — Fri, 05 Feb 2021 07:54:36 GMT

You're getting that error because your QVF file is not in the server root directory.

So, if you want to use custom path for your QVF you have to create a directory connection to your QVF file path and use it in your binary statement:

Binary lib://YourQvfPath/QvdIncrementalTest.qvf;

Re: How to reload latest QVD file only after click 'Load Data' button

mudanhong — Fri, 05 Feb 2021 08:15:14 GMT

Thanks Michele, I will do that with our Admin team.

But I tried a lot of times. Binary loading qvf file still costs a lot of time. might not be faster than Concatenating loading data from qvd file.

Do you suggest us use Binary loading than loading data from qvd file to improve the app performance?

Re: How to reload latest QVD file only after click 'Load Data' button

micheledenardi — Fri, 05 Feb 2021 08:22:10 GMT

Read data from QVD Optimized is always the fastest solution but Binary is an alternative to refresh data without read data directly from QVDs.

I believe the bottleneck of your script is during the select statement and not during Concatenate(). Right?

Re: How to reload latest QVD file only after click 'Load Data' button

mudanhong — Fri, 05 Feb 2021 09:19:39 GMT

Below is my code to load abc data: But 'Select' time almost cost much less time than loading abc_total.qvd since abc_total.qvd contains all the historical data that is huge.

Let ThisExecDate = Date(now(),'YYYY.MM.DD');
Trace ThisExecDate :$(ThisExecDate );

Total:
LOAD
*
FROM [lib://PROD QVDFiles/abc_$(ThisExecDate).qvd](qvd)
where abcname<>'123456';

Concatenate LOAD
*
FROM [lib://PROD QVDFiles/abc_Total\abc_Total.qvd](qvd)
where not Exists(abcId);

store Total into [lib://PROD QVDFiles/abc_Total\abc_Total.qvd];

drop table Total,Exclude;