Solved: Operations monitor reload task failed due to Conne... - Page 2 - Qlik Community

amithmurali · ‎2019-02-10

Dear Team,

One of our clients operations monitor failed to reload from the task scheduler. We had recently upgraded the server to November 2018 patch 1. Everything was working fine and this issue came up from past few days.

Thanks in Advance.

Please find the script log below.

2019-02-10 08:22:59 Execution started.
2019-02-10 08:22:59 Qlik Sense Server Version 12.244.6
2019-02-10 08:22:59 CPU Target x64
2019-02-10 08:22:59 Operating System Windows Server 2012 R2 Standard (64 bit edition)
2019-02-10 08:22:59 Wow64 mode Not using Wow64
2019-02-10 08:22:59 Language 0409 English English
2019-02-10 08:22:59 Country USA United States United States
2019-02-10 08:22:59 MDAC Version xxxxxxxxxxxxxxxx
2019-02-10 08:22:59 MDAC Full Install Version xxxxxxxxxxxxxxxx
2019-02-10 08:22:59 PreferredCompression 2
2019-02-10 08:22:59 EnableParallelReload 1
2019-02-10 08:22:59 ParallelizeQvdLoads 1
2019-02-10 08:22:59 EnableFlushLog 0
2019-02-10 08:22:59 UserLogfileCharset 65001
2019-02-10 08:22:59 OdbcLoginTimeout -1
2019-02-10 08:22:59 OdbcConnectionTimeout -1
2019-02-10 08:22:59 LongestPossibleLine 1048576
2019-02-10 08:22:59 WebFileTimeout 0
2019-02-10 08:22:59 WebFileUserAgent Mozilla/5.0
2019-02-10 08:22:59 DontUseInternetCache 1
2019-02-10 08:22:59 ScriptWantsDbWrite false
2019-02-10 08:22:59 ScriptWantsExe false
2019-02-10 08:22:59 ICU_FOR_CHARACTER_CLASSIFICATION true
2019-02-10 08:22:59 ICU_FOR_CODE_PAGES true
2019-02-10 08:22:59 ICU_FOR_COLLATION true
2019-02-10 08:22:59 ICU_FOR_LOCALE_INFO true
2019-02-10 08:22:59 ICU_FOR_SUPPORT_INFO true
2019-02-10 08:22:59 ICU_FOR_TIME_ZONE_INFO true
2019-02-10 08:22:59 LogFile CodePage Used: 65001
2019-02-10 08:22:59 UseMemoryFillPattern 1
2019-02-10 08:22:59 ScriptMode Standard (limits system access)
2019-02-10 08:22:59 SequentialThreadCount 8
2019-02-10 08:22:59 NumberOfCores (effective) 2
2019-02-10 08:22:59 ReloadCodebase Bnf
2019-02-10 08:22:59 Reload Executed By UserDirectory=INTERNAL; UserId=sa_scheduler
2019-02-10 08:22:59 Process Executing Qlik Sense Server
2019-02-10 08:22:59 Process ID 11144
2019-02-10 08:22:59 Grpc version 5.0.0
2019-02-10 08:22:59 Protobuf version 3.4.0
2019-02-10 08:22:59 0003 LET yr = year(ReloadTime())
2019-02-10 08:22:59 0004 SET copyright = 'Copyright 1993-2019 Qliktech International AB'
2019-02-10 08:22:59 0006 REM *** *** *** *** *** *** *** *** ****** *** *** *** *** *** *** ******
2019-02-10 08:22:59 0007 *** *** *** *** *** *** *** *** *** *** *** *** ***
2019-02-10 08:22:59 0008 *** *** *** *** *** ****** *** *** ******
2019-02-10 08:22:59 0009 *** *** *** *** *** ****** *** *** *** *** *** *** *** *** *** *** *** *** ***
2019-02-10 08:22:59 0011 SET db_v_file_override = 0
2019-02-10 08:22:59 0013 REM *** *** *** *** *** *** *** *** ****** *** *** *** *** *** *** *** *** *********
2019-02-10 08:22:59 0015 Let ReloadStartTime = now(1)
2019-02-10 08:22:59 0016 Set ahora =
2019-02-10 08:22:59 0016 SET msg =
2019-02-10 08:22:59 0016 SET skipped=0
2019-02-10 08:22:59 0016 SET loaded =0
2019-02-10 08:22:59 0016 SET textFile =
2019-02-10 08:22:59 0017 SET app_name = 'Operations Monitor'
2019-02-10 08:22:59 0018 SET app_version = '7.8.0'
2019-02-10 08:22:59 0019 Let comp = ComputerName()
2019-02-10 08:22:59 0020 LET EngineVer = PurgeChar(EngineVersion(),chr(39))
2019-02-10 08:22:59 0021 LET startMsg_1 = 'Reloading Operations Monitor 7.8.0 from AHQQLIK01 running QIX Engine version 12.244.6'
2019-02-10 08:22:59 0022 LET startMsg = 'Reloading Operations Monitor 7.8.0 from AHQQLIK01 running QIX Engine version 12.244.6. ' & If(db_v_file_override=2,'Database logs chosen',if(db_v_file_override=1,'File logs chosen','Default log source selected (will check database first, then files)'))
2019-02-10 08:22:59 0023 TRACE Reloading Operations Monitor 7.8.0 from AHQQLIK01 running QIX Engine version 12.244.6. Default log source selected (will check database first, then files)
2019-02-10 08:22:59 0023 Reloading Operations Monitor 7.8.0 from AHQQLIK01 running QIX Engine version 12.244.6. Default log source selected (will check database first, then files)
2019-02-10 08:22:59 0024
2019-02-10 08:22:59 0025 SET monthsOfHistory = 3
2019-02-10 08:22:59 0026 LET cutoffDate = AddMonths(today(1),-3,1)
2019-02-10 08:22:59 0027 Let LastReloadTime = timestamp(alt(LastSuccessfulReloadStartTime,cutoffDate))
2019-02-10 08:22:59 0028 Let lastReloadCompare = num(LastReloadTime)-1
2019-02-10 08:22:59 0030 LET serverLogFolder = 'lib://ServerLogFolder/'
2019-02-10 08:22:59 0031 LET archivedLogsFolder = 'lib://ArchivedLogsFolder/'
2019-02-10 08:22:59 0033 LET baseFileName = 'governanceLogContent_7.8.0'
2019-02-10 08:22:59 0034 LET baseTableName = 'lib://ServerLogFolder/governanceLogContent_7.8.0'
2019-02-10 08:22:59 0035 LET serviceFileName = 'governanceServiceLog_7.8.0'
2019-02-10 08:22:59 0036 LET serviceTableName = 'lib://ServerLogFolder/governanceServiceLog_7.8.0'
2019-02-10 08:22:59 0037 LET time_range_FileName = 'governance_time_range_ops_7.8.0'
2019-02-10 08:22:59 0038 LET time_range_TableName = 'lib://ServerLogFolder/governance_time_range_ops_7.8.0'
2019-02-10 08:22:59 0039 LET date_time_FileName = 'governance_date_time_ops_7.8.0'
2019-02-10 08:22:59 0040 LET date_time_TableName = 'lib://ServerLogFolder/governance_date_time_ops_7.8.0'
2019-02-10 08:22:59 0041 LET monitorAppStatsFile = 'lib://ServerLogFolder/Operations_Monitor_Reload_Stats_7.8.0.txt'
2019-02-10 08:22:59 0043 SET hideprefix = 'log'
2019-02-10 08:22:59 0044 SET firstReload = 0
2019-02-10 08:22:59 0047 SET TimeFormat = 'hh:mm:ss'
2019-02-10 08:22:59 0048 SET DateFormat = 'YYYY-MM-DD'
2019-02-10 08:22:59 0049 SET TimestampFormat = 'YYYY-MM-DD hh:mm:ss'
2019-02-10 08:22:59 0051 Let vLast4Hours = Num(timestamp(Now(1)-1/6))
2019-02-10 08:22:59 0052 Let vLast24Hours = Num(timestamp(Now(1)-1))
2019-02-10 08:22:59 0053 Let vLast72Hours = Num(timestamp(Now(1)-3))
2019-02-10 08:22:59 0055 SUB verify_database
2019-02-10 08:22:59 0140 SUB logList
2019-02-10 08:22:59 0169 SUB defineFields
2019-02-10 08:22:59 0278 SUB mappingLoads
2019-02-10 08:22:59 0303 SUB load_base_table (nombre, archivo, timestamp_field)
2019-02-10 08:22:59 0355 SUB multiNodeConfig
2019-02-10 08:22:59 0386 SUB logFolderList
2019-02-10 08:22:59 0429 SUB loadFiles (fdr,iter)
2019-02-10 08:22:59 0481 SUB load_database_logs
2019-02-10 08:22:59 0646 SUB calendarization
2019-02-10 08:22:59 0731 SUB calendarization_add
2019-02-10 08:22:59 0814 SUB concat_tables (concatToTable, incrementalTable, concatField)
2019-02-10 08:22:59 0841 SUB store_files (nombre, archivo)
2019-02-10 08:22:59 0864 SUB serviceLog
2019-02-10 08:22:59 0909 SUB reloadSummary
2019-02-10 08:22:59 0995 SUB sessionSummary
2019-02-10 08:22:59 1073 SUB exportingSummary
2019-02-10 08:22:59 1162 SUB monitor_app_stats_incremental // Use this to append new 'status' entry to table
2019-02-10 08:22:59 1176 SUB monitor_app_reload_stats (sev, obj, message, description)
2019-02-10 08:22:59 1211 SUB QRS
2019-02-10 08:22:59 1238 SUB monitor_apps_REST_user
2019-02-10 08:22:59 1320 SUB monitor_apps_REST_app
2019-02-10 08:22:59 1449 SUB monitor_apps_REST_appobject
2019-02-10 08:22:59 1543 SUB monitor_apps_REST_task
2019-02-10 08:22:59 1925 SUB supportingLogic
2019-02-10 08:22:59 1983 SUB finalize
2019-02-10 08:22:59 2022 CALL monitor_app_reload_stats('INFO','Operations Monitor', startMsg,'Reload Start')
2019-02-10 08:22:59 1178 TRACE Working on Monitor App Reload Stats
2019-02-10 08:22:59 1178 Working on Monitor App Reload Stats
2019-02-10 08:22:59 1179
2019-02-10 08:22:59 1180 IF description = 'Reload Start' THEN
2019-02-10 08:22:59 1182 IF FileSize('lib://ServerLogFolder/Operations_Monitor_Reload_Stats_7.8.0.txt') > 0 THEN
2019-02-10 08:22:59 1183 monitor_app_reload_stats:
2019-02-10 08:22:59 1184 Load * From 'lib://ServerLogFolder/Operations_Monitor_Reload_Stats_7.8.0.txt' (txt, utf8, embedded labels, delimiter is '\t', msq)
2019-02-10 08:22:59 7 fields found: Log Entry, Log Timestamp, Log Severity, Host, Description, Log Message, Object,
2019-02-10 08:22:59 20,600 lines fetched
2019-02-10 08:22:59 1185 ELSE
2019-02-10 08:22:59 1191 Let appMonitorStatsRowsInit = NoOfRows('monitor_app_reload_stats')
2019-02-10 08:22:59 1192 CALL monitor_app_stats_incremental // Add start message
2019-02-10 08:22:59 1163 Concatenate (monitor_app_reload_stats)
2019-02-10 08:22:59 1164 Load
2019-02-10 08:22:59 1165 RowNo() as [Log Entry],
2019-02-10 08:22:59 1166 timestamp(now(1)) as [Log Timestamp],
2019-02-10 08:22:59 1167 'INFO' as [Log Severity],
2019-02-10 08:22:59 1168 'AHQQLIK01' as Host,
2019-02-10 08:22:59 1169 'Reload Start' as Description,
2019-02-10 08:22:59 1170 'Reloading Operations Monitor 7.8.0 from AHQQLIK01 running QIX Engine version 12.244.6. Default log source selected (will check database first, then files)' as [Log Message],
2019-02-10 08:22:59 1171 'Operations Monitor' as Object
2019-02-10 08:22:59 1172 AutoGenerate (1)
2019-02-10 08:22:59 7 fields found: Log Entry, Log Timestamp, Log Severity, Host, Description, Log Message, Object,
2019-02-10 08:22:59 20,601 lines fetched
2019-02-10 08:22:59 1174 ENDSUB
2019-02-10 08:22:59 1194 ELSEIF description = 'Status Message' THEN
2019-02-10 08:22:59 1208 ENDSUB
2019-02-10 08:22:59 2024 CALL verify_database
2019-02-10 08:22:59 0057 TRACE Verifying logging database.
2019-02-10 08:22:59 0057 Verifying logging database.
2019-02-10 08:22:59 0058
2019-02-10 08:22:59 0059 REM *** *** *** *** *** *** *** *** *** *** *** ***
2019-02-10 08:22:59 0061 IF db_v_file_override = 1 THEN
2019-02-10 08:22:59 0072 LET db_check_time = timestamp(now(1)-0.01)
2019-02-10 08:22:59 0073 Set errormode = 0
2019-02-10 08:22:59 0075 LIB CONNECT TO 'QLogs'
2019-02-10 08:23:00 Connected.
2019-02-10 08:23:00 0078 LET tempErrorDetails = ScriptErrorDetails
2019-02-10 08:23:00 0079 IF Len(tempErrorDetails) > 0 THEN
2019-02-10 08:23:00 0093 db_check:
2019-02-10 08:23:00 0094 SELECT "id"
2019-02-10 08:23:00 0095 FROM "public"."log_entries"
2019-02-10 08:23:00 0096 WHERE "entry_timestamp" >= '2019-02-10 12:08:35'
2019-02-10 08:23:00 Error: Connector reply error: Unable to connect to database:
2019-02-10 08:23:00 Connection failed:
2019-02-10 08:23:00 Please check the values for Username‚ Password‚ Host‚ and other properties.
2019-02-10 08:23:00 0100 LET tempErrorDetails = ScriptErrorDetails
2019-02-10 08:23:00 0101 IF Len(tempErrorDetails) > 0 THEN
2019-02-10 08:23:00 0102 trace ERROR: Connector reply error: ***
2019-02-10 08:23:00 0102 ERROR: Connector reply error: ***
2019-02-10 08:23:00 0103
2019-02-10 08:23:00 0103 CALL monitor_app_reload_stats('WARN','', tempErrorDetails, 'Status Message')
2019-02-10 08:23:00 1178 TRACE Working on Monitor App Reload Stats
2019-02-10 08:23:00 1178 Working on Monitor App Reload Stats
2019-02-10 08:23:00 1179
2019-02-10 08:23:00 1180 IF description = 'Reload Start' THEN
2019-02-10 08:23:00 1195 CALL monitor_app_stats_incremental // Add status message
2019-02-10 08:23:00 1163 Concatenate (monitor_app_reload_stats)
2019-02-10 08:23:00 1164 Load
2019-02-10 08:23:00 1165 RowNo() as [Log Entry],
2019-02-10 08:23:00 1166 timestamp(now(1)) as [Log Timestamp],
2019-02-10 08:23:00 1167 'WARN' as [Log Severity],
2019-02-10 08:23:00 1168 'AHQQLIK01' as Host,
2019-02-10 08:23:00 1169 'Status Message' as Description,
2019-02-10 08:23:00 1170 'Connector reply error: ***' as [Log Message],
2019-02-10 08:23:00 1171 '' as Object
2019-02-10 08:23:00 1172 AutoGenerate (1)
2019-02-10 08:23:00 7 fields found: Log Entry, Log Timestamp, Log Severity, Host, Description, Log Message, Object,
2019-02-10 08:23:00 20,602 lines fetched
2019-02-10 08:23:00 1174 ENDSUB
2019-02-10 08:23:00 1197 ELSEIF description = 'Reload Finish' THEN
2019-02-10 08:23:00 1208 ENDSUB
2019-02-10 08:23:00 0104 tempErrorDetails =
2019-02-10 08:23:00 0105 TRACE Could not validate active database logging. Sourcing from file logs instead.
2019-02-10 08:23:00 0105 Could not validate active database logging. Sourcing from file logs instead.
2019-02-10 08:23:00 0106
2019-02-10 08:23:00 0106 SET db_v_file = 1
2019-02-10 08:23:00 0107 LET baseTableName = 'lib://ServerLogFolder/governanceLogContent_7.8.0_file'
2019-02-10 08:23:00 0108 LET lastReloadCompare = If(LastReloadSource=1,lastReloadCompare,cutoffDate)
2019-02-10 08:23:00 0110 TRACE Last Reload Compare time = 43505.467997685. CutoffDate = 2018-11-10.
2019-02-10 08:23:00 0110 Last Reload Compare time = 43505.467997685. CutoffDate = 2018-11-10.
2019-02-10 08:23:00 0111
2019-02-10 08:23:00 0111 DisConnect
2019-02-10 08:23:00 0112 SET errormode=1
2019-02-10 08:23:00 0113 EXIT SUB
2019-02-10 08:23:00 2026 REM *** *** *** ********* *** *** *** ***
2019-02-10 08:23:00 2027 CALL load_base_table ('LogContent', 'lib://ServerLogFolder/governanceLogContent_7.8.0_file','LogTimeStamp')
2019-02-10 08:23:00 0305 TRACE Checking for base qvd
2019-02-10 08:23:00 0305 Checking for base qvd
2019-02-10 08:23:00 0306
2019-02-10 08:23:00 0306 Let ts_field = '"LogTimeStamp"'
2019-02-10 08:23:00 0309 Let baseFileSize = FileSize('lib://ServerLogFolder/governanceLogContent_7.8.0_file.qvd')
2019-02-10 08:23:00 0311 IF baseFileSize > 0 THEN // Yes - QVD exists = not first load
2019-02-10 08:23:00 0313 trace Incremental reload of LogContent table (i.e. not first reload)
2019-02-10 08:23:00 0313 Incremental reload of LogContent table (i.e. not first reload)
2019-02-10 08:23:00 0314
2019-02-10 08:23:00 0314 Let firstReload = 0
2019-02-10 08:23:00 0316 LogContent:
2019-02-10 08:23:00 0317 NoConcatenate
2019-02-10 08:23:00 0318 Load * FROM [lib://ServerLogFolder/governanceLogContent_7.8.0_file.qvd] (qvd)
2019-02-10 08:23:00 0319 WHERE LogTimeStamp >= '2018-11-10'
2019-02-10 08:23:00 51 fields found: Id, LogEntryPeriodStart, LogTimeStamp, Service, Hostname, Message, Export Count, Export Store Count, Description, ProxyPackageId, RequestSequenceId, _date_time_link, _proxySessionPackage, Reload from Hub, TaskExecution, TaskId, UserId, ObjectId, ObjectName, ProxySessionId, Context, Command, Result, App Name, ProductVersion, Sequence#, qmc_change, QMC Resource Type, SecurityClass, ClientHostAddress, Severity, ActiveDocSessions, ActiveDocs, ActiveUsers, CPULoad, LoadedDocs, Selections, VMCommitted, VMAllocated, VMFree, VMPctCommitted, Cache Hits, Cache Lookups, Cache Bytes Added, Session Duration, Session Selections, Session Count, Session CPU Spent (ms), Reload CPU Spent (ms), Session KBytes Sent+Received, Reload KBytes Sent+Received,
2019-02-10 08:23:00 Error: I/O error governanceLogContent_7.8.0_file.qvd
2019-02-10 08:23:00 Execution Failed
2019-02-10 08:23:00 Execution finished.

nicolaslopedebarrios · ‎2024-10-21

In our case, qvd files were corrupted, with some data out of range. You can find out by opening the qvd file pointed out by the error with the viewer.

The issue was resolved by deleting the .qvd and letting them recreate. This worked for every error of this kind:

I/O error filenameLog_7.xx.x_file.qvd:
 
LogContent: 
NoConcatenate 
Load * FROM lib://ServerLogFolder/filenameLog_7.xx.x_file.qvd (qvd) 
WHERE LogTimeStamp >= 'yyyy-mm-dd'

Operations monitor reload task failed due to Connection error

General Question