topic Re: Application Loading and Task Scheduler in Management & Governance

Application Loading and Task Scheduler

twales — Thu, 07 Apr 2022 15:46:45 GMT

Hi,

AWS r4xlarge. Windows 2019 w/2 cores , 4 CPU's, 2/2022 AMI.

Each report is the same for 8 different DB's. The largest loads in ~30 seconds. Each is scheduled to run at the same time. There are multiple sets of reports that are similar but this is just one example.

What is happening now it seems is that 1/8 is failing due to slave not available.

I read that each task uses one core, not sure if this is true or not but the total time for loading this group should be no more than 4 minutes.

Doesn't Qlik queue each request and wait until a resource is available to run? Not sure why I need to stagger tasks when Qlik can handle it better. Otherwise were chasing report build times and scheduling based on loads and server activity.

The biggest issue forthcoming will be significantly more data per DB/report and more frequent load times using CDC.

Is there any way to pipe in recent changes, change, updates and deletes into Qlik applications without a full rebuild?

Since we can handle CDC in the cloud for all our needs we're looking for a way for Qlik reports to ingest changes without rebuild. We're not likely to invest more into software for this purpose since we already have these tools. We're looking for Qlik as is to be able to handle this capability which we think is reasonable given that CDC is decades old technology.

We're hoping for a plugin as opposed to licensing the whole Qlik Data integration suite as we have too many customized requirements. Is such a tool available?

Thanks

Re: Application Loading and Task Scheduler

rohitk1609 — Thu, 07 Apr 2022 16:54:47 GMT

This is not true: I read that each task uses one core, not sure if this is true or not but the total time for loading this group should be no more than 4 minutes.

This is true: Doesn't Qlik queue each request and wait until a resource is available to run?

Is there any way to pipe in recent changes, change, updates and deletes into Qlik applications without a full rebuild? If you are saying to update the production app, then yes you can do it, duplicate the app, make changes and publish replace.

You should have at least 4 processors. 2 is not enough

If a post helps to resolve your issue, please mark the appropriate replies as "Accept as Solution" or Like it.

Reach out to me at kumar.rohit1609@gmail.com if there is need of any clarification or assistance

Connect with me on LinkedIn https://www.linkedin.com/in/rohit-kumar-67ba152b/

To get latest updates and articles, join Qlik community at https://www.facebook.com/QlikIntellectuals

Best,
Rohit

Re: Application Loading and Task Scheduler

twales — Thu, 07 Apr 2022 18:12:55 GMT

What you noted at 4 cores required is counter to min 8GB mem on the same page you referenced. It's a moderately large deployment w/2 physical cores, 4 CPU's , 32GB mem and currently 199GB free space. I don't need 4 cores just to rebuild 8 small applications in a row which means basically I'm doubling its size in AWS world.

Qlik can't be queuing each task and waiting indefinitely for resource because the log specifically has a failure waiting for a slave resource available. The time span here is literally seconds and it fails.

The correct answer I found here which is to say Qlik isn't handling queuing beyond a few retries and then fails.

Solved: Task Priority - Qlik Community - 1909817

Basically it's overloaded which is ridiculous for the small load in discussion. A test deployment with only dev/test users, doesn't need 4 cores just to load 8 small apps in succession and if it is required then that's a Qlik problem.

Regarding CDC, no I'm not saying that I'm saying CDC which means data changes are piped into a data source in real time and application would be built in real time. It has nothing to do with duplicating applications.