Very frequent reloads on QMC

[#1]

Hi All,

I would like to understand how Qlikview Publisher works when there is a frequent reload schedule say every 10 min and reloading qvw takes 7 to 15 min to complete. Does the scheduler queued up the reload task OR cancel the scheduled task if the reload is already running.

Is there any performance implication on publisher, if this type of task is scheduled?

Thanks,

Anosh

[johnw]

I'm not a Publisher or server expert, but if I remember correctly, the newly-scheduled task fails immediately if the previous one is still running. On our system we send emails out if a task fails, so constant failures would be really annoying here, but that was our choice to send those emails. I'm not thinking of a performance implication beyond the obvious one that you'll always be burning as much CPU as it can devote to the reload task, which will depend on how many processors you have and how well it can split the load between them I believe. If you have other reloads you need to do, everything may run more slowly than it otherwise would due to competition for resources.

[#3]

Hi John, Thanks for your reply. What i saw in our QMC is that, scheduler SKIP the scheduled reload attempt if the task is already running and doesn't send failure mail as the task was not failed, only the scheduler was unable to start it.

I want to confirm the SKIP part as I don't have any way to verify this.

I am ok if one CPU thread is continuously being used by this reload but I wanted to know that there is no side effect of schedules being queued up.

[Peter_Cammaert]

You can verify which tasks had triggers firing when a previous reload was still running (or SKIP a reload as you call it) by reading Distribution Service logs.

The Distribution Service scheduler has a log of its own in C:\ProgramData\QlikTech\DistributionService. Every day a new one is created with a name like Root_YYYYMMDD.txt.

Open any log, and search for a line that says something like:

...Information TestReload.qvw: Task already started.

Best,

Peter

[#5]

Thanks Peter for your reply. What I understand from your reply is that if a previous schedule is running, the publisher skip the current schedule and just update distribution logs that "Task already started".

If this is true, then what happen if there is 64 threads of CPU and all threads are being utilized at a point of time because 64 different reloads are running in parallel. Will any new reload schedule goes into scheduler queue or just skipped because there is no CPU threads available? If the reload schedules are going into queue, will QV publisher checks if the new reload schedule is the one from already running schedules before putting it into the queue?

Regards,

Anosh

[Peter_Cammaert · Accepted solution]

I think you're mixing up a few things here. AFAIK the situation is as follows:

• Tasks: a task will not be launched again if it is already running. Tasks are a QlikView Distribution Service concept and managed entirely by the QDS. They do not have a direct relationship with CPU cores or threads, except that Qlik advises not to allow more reload engines in parallel than the number of cores minus 1. Tasks will be queued when the configured number of reload engines are busy. The first engine that becomes available will be allocated to run the next task in the queue.
• Threads are application execution paths within a single image (for example a reload engine loaded into memory) that can run in parallel on (virtual) execution units that are managed by the server cpu(s). Because QlikView is inherently multi-threaded (both the reload angine and the QlikView server), a single reload engine may launch parallel execution threads on multiple (virtual) cores. But the limitation is on reload engines, not on parallel threads. Moreover, threads follow regular multi-tasking rules as with any other service or application in a Windows server environment. They do not necessarily monopolize cpu cores.

And with respect to your last question: Yes, that's about correct. The QDS manages schedules and reload engines. When the next task is due to be executed, QDS will check whether the task is already running. If so, it will SKIP the current execution. If not, the QDS will try to allocate a reload engine. If no reload engines are free, the task will be put in the queue (first come-first served) until it ends up in first place and a reload engine becomes available.

May have been using a few words too many. Sorry.

Best,

Peter

[#7]

Thanks Peter for clarifying my doubts. I got my answer. I appreciate your valuable comments.