topic Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS in Qlik NPrinting

NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Fri, 06 Sep 2019 07:15:01 GMT

Dear Community,

Having the following set up:

- Nprinting June 2019 with two engines. Windows virtual Server 2012 R2 CPU E5-2643v2 3.5 GHz and 128 GB of RAM each one.

One of the server has NPrinting Server and Engine installed and the other has just the additional engine.

- Qlikview April 2019.

Using local connections (not sure if this issue happens with qvp, since these are much more slower), we have seen the following behaviour using this console: https://servername:4993/#/admin/plan

Sometimes, when one or more local connections are running a task, the task get stucked. Normally when is about to finish. After a while, we decide to abort the task son we can try again and continue with another one.

However, as seen in the attached snapshots, the aborted task keep having nprinting cores/resources assigned and in consecuence other tasks can't use that resources. making the enviroment unstable and much morre slower. The only way to release those resources is restarting NPrinting Scheduler Services. But obviously this is not a solution, since this issue is happening basically every day.

Please note that:

- qvw associated to this task meet all requirements and recommendations provided by qlik help (no triggers, no one value selected, minimized objects, etc). Size goes from 10 MB to 200 MB.

- Reports asociated to these task have a high number levels and cycling. Timing of succesfully execution goes from 10 to 60 minuts depending on the specific report.

We believe that high usage of CPU during report execution is impacting Scheduler service, but cant say for sure.

Any ideas?

Thank you.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Lech_Miszkiewicz — Fri, 06 Sep 2019 07:58:52 GMT

How many cpu cores you have on engine servers?
Max recommended is 12 per engine or performance will drop

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Fri, 06 Sep 2019 08:16:25 GMT

Hi Lech,

Thank you for taking the time of answering.

As recommended, we have 12 virtual proccesors on each nprinting engine machine. Snapshot attached.

Regards,

Luis.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Frank_S — Fri, 06 Sep 2019 14:31:59 GMT

Hi @luis_pimentel

You may be facing the issue with

Unsupported QV items as mentioned here: https://support.qlik.com/articles/000026081
Dedicated NP service account required: Possiblly sharing the NP service account with other NP servers or even other QlikView servers. NP service account MUST be dedicated to running a single NP server only: https://support.qlik.com/articles/000033911

If either one of or both of these are found in your system, then you will need to follow both articles to resolve the issue.

Update: I noticed your response mentions that do not have unsupported items. Please double check those shown in the article as there are other items that 'may' have been missed.

Kind regards...

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Fri, 06 Sep 2019 15:20:43 GMT

Hi @Frank_S

Thank you for your answer.

Honestly, those articles are longer than last time I read them. So we will double check both in detail again. However I still think we meet all those requirements.

Since you are from Qlik Support, are you aware of a bug or known issue with nprinting as I describe it?

What is shocking to us is that at this console: https://servername:4993/#/admin/plan we see very ofen many resources assigned to already aborted tasks. It looks that Scheduler is trying to finish a task that was already aborted (and we abort it because it was stuck).

I know my question is not very specific. But I would like to be aware of other knowledge beyond the articles you already share.

We are currently trying the following (in order to relase those cpu cores that seem stuck):

- https://support.qlik.com/articles/000055501 -> add <add key="force-unused-resolvers-closure-policy" value="1" /> to scheduler.config

and

- add <add key="qlikview-resolver-session-timeout-seconds" value="60" /> --> to scheduler.config

Finally, yes, our NP service accounts are dedicated (one for each server) and not shared with anything else. Both are also properly licensed.

Regards,

Luis.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Frank_S — Fri, 06 Sep 2019 16:31:22 GMT

Hi @luis_pimentel

Thanks for the update...it is very good to know the NP domain service accounts are unique to your NP server and QV servers.

However, unless someone from the support desk suggested changing the config file unrelated to any support article:

ie:

Your change: <add key="qlikview-resolver-session-timeout-seconds" value="60" />

I suggest changing back to the original default especially for the point above. I would also be concerned if other config changes have been made without the direction of our support or R&D teams unless a specific article is written about a specific config update. Have you made any other config file updates? If so I suggest using 'only' the default values and only changing config files recommended by our articles or directly by our R&D team. Hopefully you have a backed up copy of the original scheduler.config file.

After making the config file change to restore the default mentioned above (and any other config file changes that may have been made), I suggest:

Aborting ALL NPrinting task executions
Restart the QlikView server/Publisher server
Restart the NPrinting server
Start execution of your NP tasks.

<add key="force-unused-resolvers-closure-policy" value="1" /> change is supported as you know by article 55501 so this is ok.

Since you already appear to have the recommended number of CPU's (12) you won't need this article specifically but it might come in handy while troubleshooting if the issue persists.

NPrinting how to manage max number of QV.exe (QlikView) and QS reporting_web_renderer.exe (Qlik Sense) in use https://support.qlik.com/articles/000047616

I am not aware of a specific defect regarding this other than the points I mentioned previously but that's not to say you haven't come across something new assuming there are no undiscovered or recently inserted unsupported items into your QVW (we often find that a developer unaware of unsupported items will come along and insert them).

If the issue persists, I would recommend starting a support case so that the issue can be investigated more thoroughly.

Kind regards...

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Fri, 06 Sep 2019 18:02:28 GMT

Hi @Frank_S

Thanks for taking the time to support.

Following your advice, we just added the

<add key="force-unused-resolvers-closure-policy" value="1" />

line to the config file. No additional changes where made on this or any other config file.

We also opened a support case to investigate the issue deeper.

Regarding the article:

NPrinting how to manage max number of QV.exe (QlikView) and QS reporting_web_renderer.exe (Qlik Sense) in use https://support.qlik.com/articles/000047616

It might be useful in order to prevent a high usage of RAM consumption to crash the server or Nprinting service. We wont make that change for now, waiting for a response to the case.

I will get back to this post when having a clear view of what is happening.

Thanks again.

Luis.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Sat, 07 Sep 2019 08:21:21 GMT

Hi again @Frank_S

I am wondering If we might have an issue with the qv.exe license. Let me explain:

Our Qlikview set up is:

- 1 Qlikview Server Production (qlikviewserver1). -> Used as Access Point, and to lease licenses to users.

- 1 Qlikview Server Test License (qlikviewserver2). -> Used as ETL and for QVP connections from Nprinting.

So, even if we have properly licensed our qv.exe in both NPrinting servers agains qlikviewserver1. Using qvp connections against qlikviewserver2 might be an issue? Even if most of the times works properly.

Most of our NPrinting connections are local, but some of them are qvp against qlikviewserver2. None qvp connections against qlikviewserver1.

Attached, is an spanshot of Assigned CALs at qlikviewserver1. As you can see, both nprinting service accounts are licensed but the Last Used Date is Sep 4th (the day we licensed them). Instead of today.

Attached as well, an snapshot of the connection verification process on qlikviewserver2. If this verification is succesful, can we discard and issue with licensing?

Your inputs are highly apreciated.

Regards,

Luis.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Ruggero_Piccoli — Mon, 09 Sep 2019 14:32:36 GMT

Hi,

Did you checked the Qlik NPrinting log files? Are there any specific error?

Best Regards,

Ruggero

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Frank_S — Mon, 09 Sep 2019 14:51:37 GMT

Hi @luis_pimentel

I think there can be an issue as I personally have seen that on rare occasions.

As you most likely know, the QV test server can issue licenses but the TEST licenses are not valid for use with NPrinting as they result in a 'personal edition' message when accessed by the QV desktop that is running on the NPrinting server.

Now if you use 'dynamic' leasing on your test server I suppose in some cases there might be the opportunity that the NP service account 'leases' a test license. This is not desirable for the reason mentioned above. This will cause NPrinting to hang/fail since a valid PROD QV license is not in use. To avoid the possibility, I would recommend turning off 'dynamic' leasing on the QV test server and using manual leasing on the QV test server 'if' dynamic leasing is enabled.

'If' you are have had problems losing the valid PROD license for either of your NPrinting service accounts, you may wish to consider using a QV desktop license manually installed directly on the QV desktop on each of your NP servers (while logged on as the NP service account. (See the dedicated NP service account article I posted earlier).

Again, this is a suggested option for you whether you are losing a QV CAL or not. This suggestion would ensure that the NP service account always has a valid dedicated QV license and thus eliminating license as a possible issue.

Hope this helps...

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Ruggero_Piccoli — Mon, 09 Sep 2019 15:10:37 GMT

Hi,

To see if QlikView Desktop is running with a Personal Edition license you can:

- right click on the Qv.exe file by keeping SHIFT and CTRL pressed

- select "Run as a different user"

- insert the credentials of the user who runs the Qlik NPrinting Engine Windows service

- check that QlikView Desktop has a license activated

- repeat the same process for the user that runs the second Windows Engine service

Best Regards,

Ruggero

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Tue, 10 Sep 2019 11:55:18 GMT

Hi Ruggero,

Thank you for taking the time to follow on this.

Regarding errors in the logs, we are exactly on this scenario: https://support.qlik.com/articles/000082091 but it might not be related at all with our issue.

We do have, quite often, errors like this (with many different qvw, not just one):

Connection navigator=qlikview;documentpath=\\svmad201055\E$\Qlikview\Produccion\03.Frontal\BYLNPrintingVentasTiempoReal.qvw recorded an error in startup

and this:

Failed to open a resolver for connection navigator=qlikview;documentpath=\\svmad201055\E$\Qlikview\Produccion\03.Frontal\BYLNPrintingVentasTiempoReal.qvw. ERROR: System.AggregateException: Se han producido uno o varios errores. ---> System.Runtime.InteropServices.COMException: Error en la llamada a procedimiento remoto. (Excepción de HRESULT: 0x800706BE)↓↓ en Qlik.QlikView.Interfaces.TextObject.SetProperties(ITextObjectProperties _Properties)↓↓ en Engine.Navigator.QlikView.QV.Extensions.Extensions.TextObjectProperties.InnerApply() en C:\Jws\release-19.19.x-NOvyT9XXE\server\NPrinting\src\Engine.Navigator.QlikView\QVx\Extensions\Extensions.TextObject.cs:línea 30↓↓ en Engine.Navigator.QlikView.QV.QlikViewGraphicalObject`1.<>c.<.ctor>b__1_4(TGraphicalObjectProperties prop) en C:\Jws\release-19.19.x-NOvyT9XXE\server\NPrinting\src\Engine.Navigator.QlikView\QVx\QlikViewGraphicalObject.cs:línea 539↓↓ en Qlik.CachingProperty.Property`3.SetValue(Func`3 getValue) en C:\Jws\release-19.19.x-NOvyT9XXE\server\NPrinting\src\Qlik\Qlik.Reporting.Common\Qlik\CachingProperty.cs:línea 804↓↓ en Engine.Navigator.QlikView.QV.QlikViewGraphicalObject`1.<>c__DisplayClass11_0.<set_Show>b__0(IToggleStatus status) en C:\Jws\release-19.19.x-...

However, having this errors occur sometimes even when the task finish succesfully.

- qvw local files are placed into another server, and accesed as shown above. No network issues.

@Ruggero_Piccoli @Frank_S

A full set of log is attached on case 01713773 but we are not getting much help there. Honestly this community seems more useful.

https://support.qlik.com/QS_CaseDetails?caseId=5003z0000202B5LAAU

Thanks again.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Tue, 10 Sep 2019 12:34:21 GMT

Hi @Frank_S

These are interesting insights.

Yesterday we reviewed our QV Test Server and disabled both options (both were enabled):

- Allow license lease

- Allow dynamic CAL assignment

Both Nprinting services account keep having an assigned CAL into this test server (besides de CAL assigned into the production server). This is correct, right?

Should we remove the assigned CALs on the Test Server? It looks nprinting qvp connections still work fine with these CALs not assigned. And might be related.

However, the issue is happening right now again.

A task just got stuck with 1 pending request at https://10.10.205.23:4993/#/admin/plan and several resources assigned. Attached sreenshot.

We also made sure (again) that our client was licensing the qv.exe properly in both engines (we don't have access rights). Not 100% sure until now.

Thank you.

Luis

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Tue, 10 Sep 2019 12:35:56 GMT

Hi @Ruggero_Piccoli

we just made sure (again) that our client was licensing the qv.exe properly in both engines (we don't have access rights). Not 100% sure until now.

Thanks!

Luis.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Wed, 11 Sep 2019 06:46:45 GMT

Hi again @Ruggero_Piccoli

We have also found the following error:

Doc::GetLayout error:14 message:"Prohibit validate":

at C:\ProgramData\QlikTech\QlikView\Events_xxxx.log (attached)

This error is displayed continuosly. But we have found no info or documentation about it.

I assume is related with the .qvw itself, right? How can we investigate further this error?

Thank you again.

Luis

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Frank_S — Wed, 11 Sep 2019 14:30:26 GMT

Hi @luis_pimentel

As mentioned in one of my previous notes, I suggest that you re-check the QVW for unsupported items. If you replied with confirmation that you re-checked it my apologies. However if you did not as you suggested you would, then I suggest you re-check the QWV. It is possible that someone may have added an unsupported item or you may have missed one that is included in the documentation that I sent you. Here it is again for your convenience.

https://support.qlik.com/articles/000026081

Also, QV TEST licenses should not normally work where NPrinting is concerned. The QV desktop installed on the NP server MUST have a valid QV desktop license or a valid manually assigned CAL from the Production QV server. In this case you should take whichever action you deem best in your circumstance to ensure that separate, valid desktop license/named CALs are manually assigned to each separate NP service account in your customers environment.

https://support.qlik.com/articles/000033911

In the meantime, if you have Windows 2012 installed, you can use interactive detection service to see how the NP service account is behaving in the background for your 'stuck' task.

https://support.qlik.com/articles/000036330

IF you are running windows 2016 or higher, simply log on to the NP server as the NP service account used to run the np services on that computer and then attempt to open the QVW that is used in the connection which is getting stuck. If the NP service account has problems opening the QVW this way, you will have possibly identified that there is a problem with the QVW itself (including the possibility of an unsupported item, macro etc.

If none of the above or our previous comments helps you to resolve the issue you may be encountering some other problem. Which in that I case I would suggest to you to open a new support case with the Qlik Support desk so that the issue can be investigated more thoroughly.

Kind regards...

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Thu, 12 Sep 2019 11:43:54 GMT

Thank you @Frank_S

Regarding Interactive Service Detection, we were no able to see what was happening when the task was stuck. Good to know about this option anyway.

About having Qlikview Desktop, we also made sure is properly licensed and QV Test Server is not interfering.

Regarding the Usupported items, I did review them in detail. We met all requirements except:

- "Charts that require selections due to conditional expressions/dimensions in order to render data into the chart"

Many of our charts have conditional expressions. and conditional show in the chart itself.

This could be a reason to random stuck tasks?

- "Formula that evaluate the value of an array may not be used with NPrinting and NPrinting filters. The use of these formula with result in " - " empty set"

Due to the complexity of the report is impossible to be 100% that formulas inside levels are always evaluated <> null. Question: This applyies only to NPrinting Filters (we satisfy this for sure) or also to formulas inside the report template/levels?

Finally, we found the following error in the attached log. But there is no info about it in the community or online help.

The error seems to match with those stucked tasks.

20190911T000004.000+0200 20190911T000006.000+0200 1 300 Error Doc::GetLayout error:14 message:"Prohibit validate": 91fdc7b-c285-4ff5-9847-77c1a0bad93
20190911T000004.000+0200 20190911T000006.000+0200 1 300 Error Doc::GetLayout error:14 message:"Prohibit validate": 91fdc7b-c285-4ff5-9847-77c1a0bad93
20190911T000004.000+0200 20190911T000006.000+0200 1 300 Error Doc::GetLayout error:14 message:"Prohibit validate": 91fdc7b-c285-4ff5-9847-77c1a0bad93
20190911T000009.000+0200 20190911T000010.000+0200 1 300 Error ListBox::GetListBoxCellsEx error:14 message:"Prohibit validate": cfa2275-a8c5-420f-abb5-cd2531c338f
20190911T001509.000+0200 20190911T001511.000+0200 1 300 Error Doc::GetLayout error:14 message:"Prohibit validate": 363f17f-b40c-44ce-840d-56dbb6185f3
20190911T001509.000+0200 20190911T001511.000+0200 1 300 Error Doc::GetLayout error:14 message:"Prohibit validate": 363f17f-b40c-44ce-840d-56dbb6185f3

Do you have any input about how to investigate further this error message?

Thank you.

Luis.

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Frank_S — Thu, 12 Sep 2019 16:11:22 GMT

Hi @luis_pimentel

As per my previous detailed message...

"If none of the above or our previous comments helps you to resolve the issue you may be encountering some other problem. Which in that I case I would suggest to you to open a new support case with the Qlik Support desk so that the issue can be investigated more thoroughly."

At this point we will need to review in greater detail via a support case since there could be multiple layers of issues affecting your deployment potentially...or possibly even a defect not yet encountered.

Also if you could return here and share the root cause here (when it is finally discovered) when explored with in conjuntion with our support team via support case, it would be greatly appreciated if you could share the root cause/findings here for others in the community to benefit as well.

Kind regards...

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

Ruggero_Piccoli — Mon, 16 Sep 2019 12:24:58 GMT

Hi,

about your error I found only the following community message https://community.qlik.com/t5/QlikView-App-Development/Understanding-Log-of-QlikView/m-p/1620732 that links the help site page https://help.qlik.com/en-US/qlikview/April2019/Subsystems/Server/Content/QV_Server/QlikView-Server/QVSRM_Logging.htm.

It seems a not documented error message of QlikView. Did you updated your installation of QlikView? If not can you test a newer version? I want to exclude that it is something related to a specific QlikView version.

Then please try to run some test reports with a different QlikView connection (I mean with data from a different QWV, better if it is simpler that the one is giving you the errors) and check if you still have the same error messages. I want to evaluate if it is something related to the specific QVW.

Best REgards,

Ruggero

Re: NPRINTING ENGINE CORES ARE ASSIGNED TO ALREADY ABORTED TASKS

luis_pimentel — Mon, 16 Sep 2019 13:59:04 GMT

Hi @Ruggero_Piccoli

Thanks for your answer.

I did found that message on the community, but it doesn't help much.

Our current QV version is the latest one avalible (April 2019 SR1). And the issue occurs with many of our qvw, not just one.

After reviewing the case with Qlik Support, the issue is currently being investigated by R&D.

I will let the community know when they found an explanation.

Thank you!