Resource Control at a task level

[andile]

 Hi Support/Qlik Community.

We are running several tasks on one server and we'd like to know if there's an option of controlling the server resources for a specific task.

At a server level, the options for high and critical disk/memory usage are available and we're using those settings to control when Replicate should stop tasks should there be critical usage. However, when one particular task is using a lot of resources, it affects all tasks running on that server 'cause they'll all be stopped. What automated options are available to only stop the task that is utilizing more resources than the other tasks before we reach high/critical usage? For example, if there's 100GB disk space, disk utilization is 60GB, high usage set to 80GB and critical usage set to 90GB. One task uses 50GB of disk space while the other tasks are using just 10GB. We want to stop the task that's using more that 50% of the resources alone to avoid critical disk usage which will stop all tasks on a server impacting other tasks that are behaving normally.

Regards.

[Arun_Arasu]

Hi @andile ,

Thanks for reaching out to Qlik Community .
I am afraid if we have that feature in Qlik replicate where the individual task can be stopped based its resource usage .
Below is the user guide for your reference

https://help.qlik.com/en-US/replicate/May2023/Content/Global_Common/Content/SharedEMReplicate/Server...

In the High Disk Space Utilization Threshold section, specify the high disk space utilization threshold (in terms of percentage). When the threshold is reached, a notification will be sent (if defined).

In the Critical Disk Space Utilization Threshold section, specify the critical disk space utilization threshold (in terms of percentage). When the threshold is reached, all tasks will be stopped and a notification will be sent (if enabled). Replicate will resume the tasks automatically when there is sufficient disk space to do so.

Additionally , if you suspect that there are few tasks which normally  utilizes more resources , you can add a new notification  for that task to monitor its memory and disk usage , please refer to the below User guide for the same

https://help.qlik.com/en-US/replicate/May2023/Content/Global_Common/Content/SharedEMReplicate/Server...

Thanks & Regards

Arun

[SachinB · Accepted solution]

Hello @andile ,

We don't have such feature in replicate console where we can restrict for single task to stop when it's consuming alone more than expected threshold. However, we do have a nice feature in Qlik enterprise manager where you can monitor such task using the capacity planning.

The Capacity Planning dashboard is meant to be run for a small number of representative servers and Change Processing tasks to help you forecast sizing requirements. It provides an overview of key metrics related to activity and performance as well as providing a summary of key capacity indicator metrics.

Charts related to Server Utilization and Change Processing Performance are also shown for the given time frame. Below each chart, you can toggle to view the trending data by hour, by day, by week, by month, or by minute.
For the specified filters, the Capacity Planning dashboard displays the following information and charts:

• Overall activity
• Overall performance
• Capacity indicators
• Server utilization

Kindly refer to below link for the more information.

https://help.qlik.com/en-US/enterprise-manager/May2023/Content/EnterpriseManager/Main/Analytics/tren...

By monitoring the above dashboard, you can take call accordingly. Hope this helps.

Regards,

Sachin B

 

[andile]

Hi @Arun_Arasu, @SachinB 

Thank you for the prompt responses. I've used capacity planning and notifications for specific tasks. However, during after office hours (especially at night), the notifications are sent but can only be attended next morning. Perhaps we can look at using the QDI API to monitor utilization at task level. i.e. using the 'get_task_details', 'stop_task', and 'run_task'.

[SachinB]

Hello @andile ,

Thanks for the update.

Yes, You can expose these APIs from Qlik enterprise manager and put the condition based on response from get_task_details, Whether you want to trigger another the API which stops the task or ignore in other way.

https://help.qlik.com/en-US/enterprise-manager/May2023/Content/EnterpriseManager/EnterpriseManager_A...

Kindly test these scenarios before deploying in higher environment 

Regards,

Sachin B

[andile]

Hi @SachinB 

I'll definitely test in a non-prd environment first. Thank you.