topic Re: App reloading from the last 20 hours ?? Need help in QlikView

App reloading from the last 20 hours ?? Need help

vishalgoud — Mon, 26 Jan 2026 18:19:17 GMT

Our final application with UI is relaoding from 19-dec-2017 13:59:32 to till. almost 20 hours and still going on.

app details : earlier app size is 13 GB.

now few new columns are added to few tables so not sure about the size but definately more than 13 GB.

Prod server ram : 354 GB. 4 nodes.

For the first time facning this kind of issues Please help me.

Just explored more into log got to know that 7 left joins and 7 resident loads and 1 outer join and 1 concatination is taking more time than usual...

Re: App reloading from the last 20 hours ?? Need help

sushil353 — Wed, 20 Dec 2017 10:13:53 GMT

Cross check if there is any synthetic key or circular reference has formed due to the new colomns added.

Re: App reloading from the last 20 hours ?? Need help

vishalgoud — Wed, 20 Dec 2017 10:21:17 GMT

was verified couple of times in debug mode in my dev server no issues DM is good.

Just explored more into log got to know that 7 left joins and 7 resident loads and 1 outer join and 1 concatination is taking more time than usual...

Re: App reloading from the last 20 hours ?? Need help

marcus_sommer — Wed, 20 Dec 2017 12:58:34 GMT

If you could ensure that there are no issues with other tools/processes on the machine and that maybe nor enough RAM is available or with the OS I would assume that there is an issue with the datamodel.

I think I would run this app within the dev-environment and trace the number of records from each reload - then the same on the prod-environment and I assume that the number of records will be different probably because one of the join-statements worked properly and creates a lot of additionally records.

- Marcus

Re: App reloading from the last 20 hours ?? Need help

vishalgoud — Wed, 20 Dec 2017 13:57:17 GMT

Hi Marcus,

i can not run in dev env due to storage issue, i used to debug every time and it was fine with extractor and final app.

we want to test it in test env but due to version issue we can not as we have Qv 12 in test and Qv 11 in dev and prod.

we have many inline loads as mapping tables in the applymap. is inline load itself is a issue in Qv 12 or inline table which is used as a mapping table is a problem ?? Bit confused here.

Lot of strange issues after deployed in prod

1. fact table exctractor converted all the required qvds but not exit the script on its own, we manually killed it.

2. now the same with final UI app its reloaded for 20 hours and still going on, they killed it again.

3. now am suscpecting the generated qvds as well because of the final app behaviour.

these are giving me ache please suggest good method to deploy this in prod, Thank you in advance,

pls let me know if any info is not clear.

Re: App reloading from the last 20 hours ?? Need help

marcus_sommer — Wed, 20 Dec 2017 14:23:08 GMT

It's difficult to say how you could check for the problem. Normally the various environments should be so similar as possible regarding to the storages and the access on the data otherwise testings would have only a limited value and a move of an application between the environments will need various adjustments.

Beside this I would like above mentioned use trace and the log-files to see what's happing during the load - important are until which point runs the script normally before it stuck and how many records are loaded in which table. This should give some valuable hints which loads didn't work like expected and prevent a proper joining / (auto) concatenating with a likely resulting of huge tables and/or synthetic keys.

- Marcus

Re: App reloading from the last 20 hours ?? Need help

vishalgoud — Wed, 20 Dec 2017 17:52:23 GMT

Thanks for you response,

yeah not having same versions is across diff env is a serious problem for developers like me.

Running more time is okay we can see that in log file that it took 3 -5 hours for variuous joins but taking almost 3 hours time just before the exit script is not clear to me - is that time it is taking to save the app.

and another things is Fact extractor not ended even after generating the all the required qvds. this is an another big question for my self.

it would be great if you provide some pointers to the above, Thanks Again

Re: App reloading from the last 20 hours ?? Need help

sunny_talwar — Wed, 20 Dec 2017 18:11:05 GMT

We have seen large delays after the dashboard has finished reloading and these delays are attributed to publishing of the application. Once the dashboard finishes reloading, it is then published to another folder. This new location is what is used by Access Point to display to your users. So, based on what I told the publishing time not only includes the time to save the document, but also time to move it (and for some reason the time to move is more than the time it would take you to move it outside of publisher).

The work around we have used for a huge dashboard is to not use publisher at all and have the source location as the final location for the qvw to be used on Access Point.

Few of the issues/problems

1) The problem is that we have to be extra careful when making updates. As this is the sole copy, any changes that you save in this dashboard will straight away shown to users on access point. If the file go bad and you don't have back up, you will be in trouble.

2) Any security restriction will have to be done outside of QMC (right click into properties and security to give user access to the dashboard)

Knowing the restriction, it still might make sense to do this for one or two big dashboards as it saves on the overall time it takes to update the application with the new data.

Best,

Sunny

Re: App reloading from the last 20 hours ?? Need help

vishalgoud — Wed, 20 Dec 2017 20:16:15 GMT

Thank you sunny for such clear explanation. but 2-3 hours fine for the application which reloaded for 20 hours.

But coming to fact qvds extractor it reloaded for 6 hours and generated all the qvds but not shown successful even after 8 more hours finally we killed it manually.

as you know we were run the final UI app based on the success of Qvd extractors...any idea on why that extractor is not ending. Thanks again

My seniors are not considering all the concerns they are simply saying that some thing wrong with the script that why it is taking this much time and killed the job...

Re: App reloading from the last 20 hours ?? Need help

sunny_talwar — Wed, 20 Dec 2017 20:23:40 GMT

facts qvds extractor may not have finished because of couple of reasons (I can only think of two right now, but there might be others)

1) It is still reloading new data (doing joins, group bys etc) and there is nothing we can do about it

2) Server has gone into a weird state which require you to restart server/restart machine. This can be checked by trying to kill the reload. If you are unable to kill the reload, then it is probably in a weird state. Restarting server/restarting machine and then trying to reload again might help.

One way to get around this problem is to break down the qvd generation process into multiple qvd generators which run for smaller duration.

Fact Generator Task1

Fact Generator Task2

Fact Generator Task3

....

FinalDashboard

This will help you determine which of the fact generator qvw file is failing/not working/hanging. Additionally, you can parallel load some/all of them in order to save on some reload time if they are not dependent on the previous task.

Re: App reloading from the last 20 hours ?? Need help

vishalgoud — Wed, 20 Dec 2017 21:44:23 GMT

Thanks again,

I have to add one more point here, in my fact extractor - it will exctract latest 2 months data and in the final tab i wrote one forloop which can be used to load the old qvds and add few new columns with default values to it and store them in a different path with the same name..

that loop i wrote for 3 diff qvds after this i wrote exit script. PFB.

was suspecting the this loop is not ending there before exit script. that is the reason it is not ending.

YearData:
LOAD
DISTINCT
DATE(Date#(SubField(FileBaseName(),'_',3),'YYYYMMDD'),'YYYYMMDD') AS YrMonth
From (qvd);

let cnt=FieldValueCount('YrMonth');

FOR i =1 to cnt;
LET vYr_Split_StoreQVD = FieldValue('YrMonth',i);

NoConcatenate
SBP_TEST_$(vYr_Split_StoreQVD):
LOAD
*,
0 as NoticePeriod,
0 as Workinghours,
0 as Rating

From (qvd);

STORE SBP_TEST_$(vYr_Split_StoreQVD) into E:\Development\ESI\MBP Test\Old Qvds\SBP_TEST_$(vYr_Split_StoreQVD).qvd(qvd);

DROP Table SBP_TEST_$(vYr_Split_StoreQVD);

Next i;

DROP TABLES YearData;

EXIT Script;

Re: App reloading from the last 20 hours ?? Need help

sunny_talwar — Wed, 20 Dec 2017 23:05:25 GMT

First thing to do is to add some TRACE statements to make sure you can see the progress in the log files.... Also, not sure if it makes any difference, I use $(i) inside FieldValue instead of just i..

YearData:
LOAD
DISTINCT
DATE(Date#(SubField(FileBaseName(),'_',3),'YYYYMMDD'),'YYYYMMDD') ASYrMonth
From (qvd);

LET cnt = FieldValueCount('YrMonth');
TRACE $(cnt);

FOR i =1 to cnt;
LET vYr_Split_StoreQVD = FieldValue('YrMonth', $(i));

TRACE $(vYr_Split_StoreQVD);

Next i;

DROP TABLES YearData;

EXIT Script;

And once you see that nothing looks wrong above, try this

YearData:
LOAD
DISTINCT
DATE(Date#(SubField(FileBaseName(),'_',3),'YYYYMMDD'),'YYYYMMDD') ASYrMonth
From (qvd);

LET cnt = FieldValueCount('YrMonth');
TRACE $(cnt);

FOR i =1 to cnt;
LET vYr_Split_StoreQVD = FieldValue('YrMonth', $(i));

TRACE $(vYr_Split_StoreQVD);

NoConcatenate
SBP_TEST_$(vYr_Split_StoreQVD):
LOAD
*,
0 as NoticePeriod,
0 as Workinghours,
0 as Rating

From (qvd);

STORE SBP_TEST_$(vYr_Split_StoreQVD) into E:\Development\ESI\MBP Test\Old Qvds\SBP_TEST_$(vYr_Split_StoreQVD).qvd(qvd);

DROP Table SBP_TEST_$(vYr_Split_StoreQVD);

Next i;

DROP TABLES YearData;

EXIT Script;

Re: App reloading from the last 20 hours ?? Need help

vishalgoud — Thu, 21 Dec 2017 11:53:19 GMT

Thanks sunny ,

Didnt get a chance to look into that Fact extractor again, But was reloaded the same final app(UI) in test env by poinitng that to prod qvds which we generated using the fact exctracotr...

Guess what now its reloaded in 50 Mins in test env with the same qvds.

but in prod it took more than 20 hours....Dont know how many surprises i will get in future , with this now things are bit more complicated..

Any pointers on this please..