How to Decrypt Qlik Replicate Verbose Task Log Files

[john_wang]

Environment

• Qlik Replicate November 2021 (Version 2021.11 and later)

What is Replicate log file encryption?

Starting from Qlik Replicate version "November 2021", when the logging level is set to Verbose, sections in a Replicate log file that might contain customer data will be encrypted while the rest of the log will remain in clear text, this is implemented by the enable_data_logging configuration parameter set to true by default in order to prevent customer data snippets from appearing in verbose logs.

The sample line of the task log file looks like this:

2021-11-05T15:45:11:390848 [AT_GLOBAL ]I: Logging of database data is disabled (at_logger.c:2702)

However, in some support cases, R&D may require the ability to decrypt the log in order to analyze the problem. In such cases, Qlik Support will ask you to provide the task or environment-level encryption key and corresponding task log files:

• Replicate task diagnostics package file, and
• log.key file for decrypting a task's log file, location is <REPLICATE_INSTALL_DIR>\data\tasks\<task-name>
• log.key file for decrypting non-task log files (e.g. server logs), location is <REPLICATE_INSTALL_DIR>\data

Below is the detailed steps how to enable the database data logging and decrypt a task log file with corresponding log.key files.

Steps

1. Stop the Replicate Server service.
2. Open <REPLICATE_INSTALL_DIR>\bin\repctl.cfg and set the enable_data_logging parameter to true. (add the line if it does not exist). This is set to true by default in version 2021.11 and later, even if you do not see this line in your configuration file.
   {
   "port": 3552,
   "plugins_load_list": "repui",
   ... ...
   "enable_data_logging": true
   }
   
3. Save the repctl.cfg file and start the Replicate Server service.
4. Now the task log files contain encrypted database data, sample lines like:
   
   

   2021-11-05T15:59:07:969146 [AT_GLOBAL ]I: Log file encryption base "755opF/9oGw7YOx+2ma0dA==" (at_logger.c:2695)
   2021-11-05T15:59:07:969146 [INFRASTRUCTURE ]I: The log level for 'TARGET_APPLY' has been changed from 'INFO' to 'VERBOSE'. (at_logger.c:2927)
   2021-11-05T15:59:07:969146 [INFRASTRUCTURE ]I: The log level for 'SOURCE_CAPTURE' has been changed from 'INFO' to 'VERBOSE'. (at_logger.c:2927)

   ... ...

   2021-11-05T16:02:21:346746 [SOURCE_CAPTURE ]V: Event of captured table 'SCOTT.KIT' (data object id '93719', object id '93719') (oradcdc_parse.c:303)
   2021-11-05T16:02:21:346746 [SOURCE_CAPTURE ]V: xid [000008fb00100007] (oradcdc_parse.c:166)
   2021-11-05T16:02:21:346746 [SOURCE_CAPTURE ]V: BI col 2 [5]
   ~uAAAABvF/duclauaOYlVdTbsCpbAiSZGsq3DIp8XyImAZjY984KVgc62Lir20Mnw1VU1V/54j/vXa4BXpyQ7rjcEPR1MtjoYnpPSeu4QSwfiUx3l}~ (oradcdc_parse.c:340)

   The latest line is the encrypted database data (BeforeImage of an update).
   
   
5. Run "Qlik Replicate Command Line" prompt "as administrator", and change to directory <REPLICATE_INSTALL_DIR>\bin
6. Run the following command to decrypt task logs:
   
   repctl dumplog <log-file-path> <log.key-path> [> <path-to-output-file>]
   
   A sample command:
   
   repctl dumplog "C:\Program Files\Attunity\Replicate\data\logs\reptask_Ora-to-SQL-2.log" log_key="C:\Program Files\Attunity\Replicate\data\tasks\Ora-to-SQL-2\log.key" > "C:\Program Files\Attunity\Replicate\data\logs\reptask_Ora-to-SQL-2-dump.log"
   
   Now we see the decrypted data in the task log files. The latest line is the BeforeImage of an update in clear text (in this sample it is text string "xxxxx"):
   
   2021-11-05T16:02:21:346746 [SOURCE_CAPTURE ]V: Event of captured table 'SCOTT.KIT' (data object id '93719', object id '93719') (oradcdc_parse.c:303)
   2021-11-05T16:02:21:346746 [SOURCE_CAPTURE ]V: xid [000008fb00100007] (oradcdc_parse.c:166)
   2021-11-05T16:02:21:346746 [SOURCE_CAPTURE ]V: BI col 2 [5]
   25a058b66d8: 7878787878 | xxxxx
   (oradcdc_parse.c:340)

Related Instructions

The encryption key file (log.key file) can be used to decrypt any log that was encrypted with it, not just the log for which Qlik Support requested the encryption key file. If this is a concern, you can generate a log.key file that will only be valid for the specific log requested by Qlik Support.

To do this:

1. Delete the log.key file.
2. Restart the task or Replicate Server service (depending on the log type required) to generate a new log.key file.
3. Send the requested log file and the log.key file to Qlik Support.
4. Delete the log.key file.
5. Repeat steps 2-4 if you need to provide additional encrypted logs.

If you do not want verbose logs encrypted, open <REPLICATE_INSTALL_DIR>\bin\repctl.cfg and set the enable_data_logging parameter to false then restart the Replicate server service. Add the line if you do not see it in the file.

Products: Qlik Replicate 

[john_wang]

If you want to create "log.key" file manually, see article

Replicate - Generate a log.key file manually  

[nareshkumar]

I am getting following error in my linux environment :

ran the command from install bin :

[root@dc2-uvlqlkq01 bin]# ./repctl dumplog "/QlikApp_Stage/qlikqa/logs/reptask_LIOSPG_LCADM_MDW_LS.log" "/QlikApp_Stage/qlikqa/tasks/LIOSPG_LCADM_MDW_LS/log.key" > "/QlikApp_Stage/qlikqa/logs/reptask_LIOSPG_LCADM_MDW_LS_dump.log"
./repctl: error while loading shared libraries: libarepbase.so: cannot open shared object file: No such file or directory
[root@dc2-uvlqlkq01 bin]

[Dana_Baldwin]

Hi @nareshkumar 

Please try running the arep_login.sh script first to ensure all the paths to the libraries are set:

source /opt/attunity/replicate/bin/arep_login.sh

Thanks,

Dana

[Steve_Nguyen]

Notes, if you try to Decrypt and your decrypted log show :

[dumplog command] Failed
command dumplog: Redundant positional argument (exit status 22)

then you need to add qualifier for the log.key

example:

~\bin>repctl dumplog "C:\deceptask_sql-2-sql.log" log_key="C:\temp\log.key" > "C:\temp\decrypted.log"