Changing Central Node to Slave results in Monitoring Apps not showing Node logs

[duzunic88]

Hi,

Recently I made a change to our environment, making the central node scheduler the Slave.  This was done to guarantee that the central always handle all reloads while the other node which is consumer facing and as well the failover candidate (failover must have scheduler enabled) was made the Master.  While this works great I noticed that all my monitor apps now only show logs information for the central node and not the consumer node. On top of that, the monitor apps have no prior historical data for the central, just starting on the date when I made the Slave/Master change. 

I'm curious why the Slave/Master change impacted monitoring apps?  I'm not sure why making the central node handle all reloads, would impact what data it retrieves from the ArchivedLogs folder which does indeed contain both the nodes data.  Like I said this was all working fine until the Slave/Master change.  Any help would be appreciated.

[duzunic88]

@Bastien_Laugiero see below:

When you say this file was not updated since 12/26, is this file stored in C:\ProgramData\Qlik\Sense\Log on the rim node or on the central node?

I'm looking on the central node(219 in my case, screenshot attached) located at C:\ProgramData\Qlik\Sense\Log.  Just to check, governanceLicenseLog_7.10.0_db.qvd on one of the rim nodes also reads last date at 12/26/2018.

If you reload the application called Log Monitor, you should see this file being generated/updated on the central node.

Log monitor was last refreshed on 1/08/2019 so it seems like it is not updating the _file.qvd on the central? (log attached from 1/08/2019)

Quick question, when you used SET db_v_file_override = 1, do you see the data for both nodes or still only for the central node?

I do see the rim nodes show up and they do have data prior to the date mentioned, but nothing for Jan 2019 (users definitely hit those nodes, central is only for reloads, 230 and 245 are consumer nodes.) Screen shot attached.

Thanks! 

[Bastien_Laugiero]

Hello,

Thanks for the script log for the LogMonitor. I can see in this logs that the variable db_v_file_override = 0 so you are loading the data from the Qlogs database on this application. 

I would suggest to set it to 1 as for the other monitoring application to load your data from log files. 

So the reason why the file GovernanceLogMonitor_7.10.0_file.qvd is not updated is that it's still the file GovernanceLogMonitor_7.10.0_db.qvd that gets updated and we can see that from the screenshot 219 central node.png.

The screenshot 230 and 245 Rim Nodes for Jan.PNG is from the license monitor app I guess. What is interesting here is that on the screenshot 219 central node.png we see the _file.qvd associated with that application being updated. (for example GovernanceLicenseLog_7.10.0_file.qvd)

Would it be possible to send the script log for the License Monitor app to see why it's not loading the data for January?

Hope that makes sense!

Thanks!

[duzunic88]

@Bastien_Laugiero

Attached!

I do want to bring up something that I cam across while digging pretty deep into the operations logs (achieved by setting db_v_file_override = 1 for operations monitor).  On the attached (Operations Log Monitor screenshot) we can see the repository having issues communicating with the nodes and the timestamp matches up when the problems started.. Could this be my culprit? I wonder what could cause this, maybe a windows update?

[Bastien_Laugiero]

Hello, 

Thanks for the logs and screenshot. 

From the log, I can see that the License Monitor is still using db_v_file_override = 0 (so loading from the Qlogs database and not the log files)

I wanted to check the logs for an app that has db_v_file_override = 1 to see which log file it can load and which it cannot.

Could you either set the license monitor with  db_v_file_override = 1, reload and attach the logs or attach the reload logs of the operation monitor which seem to be configured with db_v_file_override = 1?

As for your observation, yes it looks like Qlik Sense experienced issue to move the logs in the archived log folder. 

Quick question, if you go manually in the archived logs folder for the rim node. Do you see logs in there that are after December 26th?

Thanks! 

[duzunic88]

@Bastien_Laugiero

Could you either set the license monitor with  db_v_file_override = 1, reload and attach the logs or attach the reload logs of the operation monitor which seem to be configured with db_v_file_override = 1?

Attached!

Quick question, if you go manually in the archived logs folder for the rim node. Do you see logs in there that are after December 26th?

Attached are two images.  The 245box.png is showing C:\ProgramData\Qlik\Sense\Log from the actual rim node server, which do confirm that the _db qvds have not been updating since 26/27th.

The second one is from D:\Qlik Data\ArchivedLogs\245 which is the actual ArchivedLog directory for 245 on the central node.. not much here except individual folders so not sure if you meant something else? 

THANK YOU!!

[Bastien_Laugiero]

Hello,

So now that you have changed the license monitor to load from log files, I can see it loading data for January. 

So in the license monitor at least, you should see data for January. Can you confirm? 

Thanks!

[duzunic88]

So it does indeed look like Jan log data is coming in but only for 219.  Please see attached.  When I select Jan 2019 and the the two consumer nodes (230 and 245), they show no session information even though about 500 users log in every day between those two servers 😞

[duzunic88]

So it does indeed look like Jan log data is coming in but only for 219.  Please see attached.  When I select Jan 2019 and the the two consumer nodes (230 and 245), they show no session information even though about 500 users log in every day between those two servers 😞

[Bastien_Laugiero]

Hum interesting. 

At this point, I would suggest opening a support case because it might need to have a deeper look into your environment, log files and so on.

Sorry that we could not completely solve it but I would be glad if you can share the solution once support figured it out. 

Thank you for all your testing and patience! 🙂 

[duzunic88]

Yeah at this point I would assume it issue with the nodes communicating to the central or our installation of Qlik has some sort of issue.. will log case and let's see where it goes!

THANK YOU so much for your help, will let you know what happens!