QMC affected by numbers of fiels in EDXResult

[waszcma1]

Hi,

I have a problem with QlikView May 2021 SR2

There is something wrong with synchronization of the services and xml files generation in EDXResult folder

Week ago I discovered that non of the task can be triggered either manualy or by scheduler they seems to be frozen when I pressed the run it took couple second to proccess the action and back to prevoious stage without any logs.

After checking forum I have found that somone is recommending clear the folder C:\ProgramData\QlikTech\DistributionService\EDXResult from older files but system was constantly frozen when I tryied to entry to that folder, IT said that there must be incredible high number of files even more then 200k or 1 mln,

so we delete the whole EDXResult folder and create empty one with the same name -it works QMC start working as normal

but checking this folder I see every day increase in number of files over 26000 a day ! it is not acceptable as after couple of days we have more than 100 000 fiels and the same problems comming.

What do you recommend to do as a first action? what could be a root cause of generation so many files every day?

p.s.
After when I delete the folder and run only one small task which normaly takes 35 sek it generate 100 + fiels in after when task was completed.

and most of this files looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<Root FinishTime="" StartTime="" TaskLogFilePath="" TaskStatus="Waiting" TaskId="0a254874-5bf7-4f56-a0cc-e613ab3cee51" ExecId="00019809-6230-4d83-86fa-15063cf31d43"/>

[marcus_sommer · Accepted solution]

AFAIK each task-execution will be logged there and by using a multi-tier data-architecture with a high degree of divisions of work and a quite frequently scheduling the number of log-files per day could become very large.

I'm quite sure that we did not adjust any configuration in this regard - what means we are using the default-settings - we see there only the files of the last 30 days. Further I doubt that these 30 days are hard-coded within the distribution engine else they are probably set anywhere within the config-files - which means it should be possible to reduce the number of kept days. Just writing this I looked within the qvdistributionservice.exe.config and find for example these entries:

<!-- The number of days to keep EDX Result Files -->
<add key="NbrOfDaysToKeepEDXResultFiles" value="30" />
<!-- The number of days to keep QDS logs -->
<add key="NbrOfDaysToKeepQDSLogs" value="30" />

and maybe there are more related settings.

Beside this you may also consider to use any batch-file which looked daily for the files and moved / deleted the older ones - maybe after reading + saving the content within any qvd's whereby I doubt that it would be really sensible to keep these information for a longer period of time.

- Marcus

[waszcma1]

But forcing to deleting the xml fiels from EDXResult folder want affect working of QMC sheduler?

I used bat file to delete every night the files but some users rised the issue that they were not able to run newly created tasks as they have status "unavailable"

I have tryied 

It didnt help 

So  I hade to restart whole server and then it fix the problem. But I am not sure if this is related to EDXResults configuration files or it was just some other issue.

The thing that worries me the most, why suddenly the server start generating so high number of daily xmls files?

Could be a cause that we start using publishing to the Qlik Saas?

[marcus_sommer]

I don't know it but I wouldn't be surprised if the distribution engine has any further access to these files and/or they have some references within another log-files which may cause the observed "unavailable" status - maybe just cached for some time. To avoid it you may consider to stop the service before accessing the files and starting it afterwards again - this could be included within the batch. Of course not multiple times during the working hours but once a day within the night ...

If your issue started recently without bigger changes to the reload-tasks it seems quite likely that the distribution tasks to the SaaS is also logged in this way and now producing this high number of files.

Personally I would try to set the above mentioned settings to one or two days to see if it worked. Further I would look if there are other settings which may influence the behaviour - for example if the reload-tasks and the distribution tasks might be configured/enabled differently. And the external batch-force would be the worst case solution.

- Marcus

[Bill_Britt]

Hi,

Your best bet is to stop the Distribution service and Management service, then rename the DistributionService. Once that is done restart the Distribution service and Management service and that should rebuild the files and folders. 

Then as @marcus_sommer  stated set the values to auto-delete the files

<!-- The number of days to keep EDX Result Files -->
<add key="NbrOfDaysToKeepEDXResultFiles" value="30" />
<!-- The number of days to keep QDS logs -->
<add key="NbrOfDaysToKeepQDSLogs" value="30" />

[waszcma1]

<add key="NbrOfDaysToKeepEDXResultFiles" value="30" />  did a job 

but still actual days + previus day generate ~ 100 000 files which is managable by windows but still unknow for me why there are so many files?

Seems that 99% of those files are just about waiting status 

[Bill_Britt]

Hi,

This could be caused by the task being in a queue state. I suspect that the QDS service was restarted. So, it left those in that state.

[waszcma1]

I dont think so this is the case as we dont have such status very ofthen, we have some tasks which are running cupple hours but the number of parrarel tasks are always in the limits.

The only thing which is wondering me a litle could it be so because we use publishing apps into Qlik SaaS?

But I have also checked this and picked some random tasks ID and check which task were running and seems that all tasks even those which are used only for pure data extraction and triggering next transformation tasks are in the EDX logs.

So it seems all of our tasks are reported into logs with "waiting" status which frankly saying seems not be true. Most of them are running as normal.

Of course there are some tasks in error but this is nothing new before the EDX issue all tasks were running very similar 

[marcus_sommer]

Maybe there is a similar behaviour by the publishing-tasks like by the reload-tasks which track the task-status during the reload. If any load-statements needs more time as a certain threshold like it happens by larger aggregations/joins or by a slow data-base connections it leads to an increasing of the threshold to avoid too large task-logs. In regard to the EDX-logs this may not be properly applied and then resulting in many log-files. Just an idea to possible causes ...

...

(17.08.2022 06:11:57) Information: Reloading.

(17.08.2022 06:11:58) Information: Reloading..

(17.08.2022 06:11:59) Information: Reloading...

(17.08.2022 06:12:00) Information: Reloading....

(17.08.2022 06:12:01) Information: Reloading.....

(17.08.2022 06:12:02) Information: Reloading......

(17.08.2022 06:12:03) Information: Reloading.......

(17.08.2022 06:12:04) Information: Reloading........

(17.08.2022 06:12:05) Information: Reloading.........

(17.08.2022 06:12:06) Information: Reloading..........

(17.08.2022 06:12:07) Information: Slow down logging. Log every <2> seconds.

(17.08.2022 06:12:07) Information: Reloading

(17.08.2022 06:12:09) Information: Reloading.

(17.08.2022 06:12:11) Information: Reloading..

(17.08.2022 06:12:13) Information: Reloading...

(17.08.2022 06:12:15) Information: Reloading....

(17.08.2022 06:12:17) Information: Reloading.....

(17.08.2022 06:12:19) Information: Reloading......

(17.08.2022 06:12:21) Information: Reloading.......

(17.08.2022 06:12:23) Information: Reloading........

(17.08.2022 06:12:25) Information: Reloading.........

(17.08.2022 06:12:27) Information: Reloading..........

(17.08.2022 06:12:29) Information: Slow down logging. Log every <10> seconds.

(17.08.2022 06:12:29) Information: Reloading

(17.08.2022 06:12:39) Information: Reloading.

(17.08.2022 06:12:49) Information: Reloading..

(17.08.2022 06:12:59) Information: Reloading...

(17.08.2022 06:13:09) Information: Reloading....

(17.08.2022 06:13:19) Information: Reloading.....

(17.08.2022 06:13:29) Information: Reloading......

(17.08.2022 06:13:39) Information: Reloading.......

(17.08.2022 06:13:49) Information: Reloading........

(17.08.2022 06:13:59) Information: Reloading.........

(17.08.2022 06:14:09) Information: Reloading..........

(17.08.2022 06:14:19) Information: Slow down logging. Log every <60> seconds.

(17.08.2022 06:14:19) Information: Reloading
....

- Marcus

[KHSDM]

I'm facing the same issue... but in my case, i have millions of EDX files. (around 300k files per day)