NPrining Scheduler stopping automatically and Engine status is offline

[khan_imran]

Hello All,

I have upgraded to Nprinting Nov'18 version and scheduler services are getting stopped automatically. I am getting below error which i took from nprinting_scheduler log file.

Also Nprinting Engine status is not changing to Online from Offline.

Server OS is Windows Server 2016 Standard

QlikView Server Version is 12.1

Failed 3 times to prepare queue with exchange name engine_exchange and queue name engineallocationstatus_queue. ERROR: System.TimeoutException: The operation requested on PersistentChannel timed out↓↓   at EasyNetQ.Producer.PersistentChannel.InvokeChannelAction(Action`1 channelAction)↓↓   at Qlik.NPrinting.Common.EasyNetQ.NPrintingClientCommandDispatcherThreaded.Invoke(Action`1 channelAction, CancellationToken cancellationToken) in C:\Jws\release-18.39.x-w104zyghT\server\NPrinting\src\Common\EasyNetQ\NPrintingClientCommandDispatcher.cs:line 136↓↓   at Qlik.NPrinting.Common.EasyNetQ.NPrintingClientCommandDispatcherThreaded.Invoke(Action`1 channelAction) in C:\Jws\release-18.39.x-w104zyghT\server\NPrinting\src\Common\EasyNetQ\NPrintingClientCommandDispatcher.cs:line 174↓↓   at EasyNetQ.RabbitAdvancedBus.ExchangeDeclare(String name, String type, Boolean passive, Boolean durable, Boolean autoDelete, Boolean internal, String alternateExchange, Boolean delayed)↓↓   at Qlik.NPrinting.Common.Messages.RabbitTopologyBase.InternalPrepareQueue(String exchangeName, String queueName, Boolean useHashAsRoutingKey, Boolean queueDurability, Nullable`1 maxPriority, IExchange& exchange, String& routingKey) in C:\Jws\release-18.39.x-w104zyghT\server\NPrinting\src\Common\Messages\RabbitTopologyBase.cs:line 72

Kindly assist me to solve this problem.

Regards,

Imran K

[Gysbert_Wassenaar]

I've found this support article which recommends to uninstall and reinstall the Nprinting Engine.

[Bastien_Laugiero]

Hello,

What you are describing is often caused by RabbitMQ (NPrinting Messenger service) not being fully operational. 

Please take a look at the steps described in those two articles:

• How to resolve NPrinting Scheduler Windows Service Crashing Shutdown
• RabbitMQ/Erlang Troubleshooting Guide

Hope this helps. 

Kind regards, 

[deec]

Was there ever a resolution for this issue? I've tried all of the recommendations listed below in @Bastien_Laugiero's message. I'm close to opening a support ticket for this, but wanted to check here first.

I have the exact same issue when I install a fresh copy of November 2018 Server. Approximately 5 min after installation, one of the 4 services will stop (Scheduler service)
I can uninstall and reinstall other versions without issues on this same server (have tested Feb 2018 initial and Sept 2018 SR2).

The error presents itself shortly after the Server file installation. While the Engine doesn't need to be installed for this error to occur, it will be affected. Since the error effectively takes out the Scheduler service, engine will never be listed 'Online'.

I'm attaching a log that shows the Sept 2018 installation didn't have this problem. As soon as it upgrades to the Nov 2018, the issue starts. Proper uninstall and restart before the change.

[Bastien_Laugiero]

Hello,

Sorry for the delay and thank you for that information. 

If the scheduler stops, it could be that NPrinting Messaging Service is not functioning properly. 

Could you log on as the NPrinting service account and run the following command in Command Prompt with NPrinting November 2018: 

cd C:\Program Files\NPrintingServer\rabbitmq_server-3.7.7\sbin
rabbitmqctl.bat list_permissions

If everything works you should expect something like this as result. 

Listing permissions for vhost "/" ...
client_engine   .*      .*      .*
remote  .*      .*      .*
client_scheduler        .*      .*      .*
client_webengine        .*      .*      .*

Thank you! 

[deec]

Thanks for the response @Bastien_Laugiero

After restarting the services, the output (rabbitmqctl.bat list_permissions) is exactly as you listed.

As mentioned in my earlier message, I have tried everything in both of your earlier troubleshooting links.

I believe it is a rabbit issue as well. I've gone down a rabbit hole (no pun intended) on the internet trying other solutions. One thing I did not try is adding to the config rabbit file as suggested here: https://github.com/sensu/sensu-transport/issues/58

I was searching for the "cipher" issue after looking at the rabbit log and seeing this:

2019-01-14 09:42:56.387 [error] <0.318.0> Supervisor rabbit_epmd_monitor_sup had child rabbit_epmd_monitor started with rabbit_epmd_monitor:start_link() at <0.669.0> exit with reason {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-434907915","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen..."},...]},...]} in context child_terminated
2019-01-14 09:42:56.821 [info] <0.721.0> TLS server: In state hello at tls_handshake.erl:197 generated SERVER ALERT: Fatal - Insufficient Security - no_suitable_ciphers

2019-01-14 09:43:56.394 [error] <0.720.0> ** Generic server rabbit_epmd_monitor terminating 
** Last message in was check
** When Server state == {state,{erlang,#Ref<0.1470874218.2275934211.222120>},erl_epmd,"rabbit","SERVERNAME",25672}
** Reason for termination == 
** {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-577794225","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-01-14 09:43:56.394 [error] <0.720.0> CRASH REPORT Process rabbit_epmd_monitor with 0 neighbours crashed with reason: {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-577794225","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen..."},...]},...]}
2019-01-14 09:43:56.394 [error] <0.318.0> Supervisor rabbit_epmd_monitor_sup had child rabbit_epmd_monitor started with rabbit_epmd_monitor:start_link() at <0.720.0> exit with reason {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-577794225","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen..."},...]},...]} in context child_terminated
2019-01-14 09:44:56.422 [error] <0.725.0> ** Generic server rabbit_epmd_monitor terminating 
** Last message in was check
** When Server state == {state,{erlang,#Ref<0.1470874218.2275934209.215472>},erl_epmd,"rabbit","SERVERNAME",25672}
** Reason for termination == 
** {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-374938085","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-01-14 09:44:56.422 [error] <0.725.0> CRASH REPORT Process rabbit_epmd_monitor with 0 neighbours crashed with reason: {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-374938085","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen..."},...]},...]}
2019-01-14 09:44:56.422 [error] <0.318.0> Supervisor rabbit_epmd_monitor_sup had child rabbit_epmd_monitor started with rabbit_epmd_monitor:start_link() at <0.725.0> exit with reason {enoent,[{erlang,open_port,[{spawn_executable,false},[{args,["-sname","epmd-starter-374938085","-noshell","-eval","halt()."]},exit_status,stderr_to_stdout,use_stdio]],[{file,"erlang.erl"},{line,2213}]},{rabbit_nodes_common,ensure_epmd,0,[{file,"src/rabbit_nodes_common.erl"},{line,76}]},{rabbit_epmd_monitor,check_epmd,1,[{file,"src/rabbit_epmd_monitor.erl"},{line,108}]},{rabbit_epmd_monitor,handle_info,2,[{file,"src/rabbit_epmd_monitor.erl"},{line,79}]},{gen_server,try_dispatch,4,[{file,"gen..."},...]},...]} in context child_terminated

Here are some of the things I have tried:

- ensure erlang.cookie is present in correct directory
- Remove and reinstall RabbitMQ (cmd rabbitmqctl.bat)
- Check System Environment Variables (ERLANG_HOME)
- Enable TLS 1.0 via RegEdit
- Checked .NET framework 4.6.2
- Uninstalling Server and Engine. Reinstalled Server only (Nov2018).
- Enable TLS 1.1, SSL 3.0 via RegEdit --REVERTED

-Added Inbound/Outbound TCP/UDP allow rules for port 15672 --REVERTED
-Checked that erl.exe is the application listening on the 5672 port

Finally just gave up and tried installing other versions (Feb, Sept) and had no issues getting them to run successfully.

[Frank_S]

Hello,
If the messaging services does not start, then you may wish to review and follow the steps in the following article to resolve the issue.
https://qliksupport.force.com/articles/000043782

[deec]

thanks for the suggestion @Frank_S, however, the Messaging Service is running.

This is a fresh server capable of running Feb or Sept 2018 Nprinting without this issue. This leads me to believe that there may be a problem with the newer version of rabbit being used in the November 2018 release. (Rabbit 3.7.7). Or a problem with my server's compatibility with the new rabbit version being used.

[Frank_S]

The steps provided in my previous post may still be useful if the messaging service is damaged in some way...even if it is still starting up normally as a windows service.

I would recommend following those steps as they will repair a potentially damaged rabbit db in your environment.

In my humble opinion, if suggested article does not solve the issue, then I suggest submitting a support request so that the issue can be investigated in greater detail.

[deec]

Thank you for your suggestions. Unfortunately the problem is not resolved after deleting the rabbitMQ folder and restarting the services as the last suggestion instructed. 

At this point, I see no option but to open a support ticket if I want to use November 2018

If anyone else has suggestions I may try. If anyone resolves the issue please update.