topic Re: How do I know whether a task is full load or CDC using the metadata JSON? in Qlik Replicate

How do I know whether a task is full load or CDC using the metadata JSON?

S_Lo_DE — Wed, 10 Jan 2024 20:49:12 GMT

Hi there,

I have the task metadata JSON generated but I cannot find a field in task_settings to indicate whether a task:

1) Full Load or CDC

2) Is enabled or not(Should I look at scheduler instead?)

Is there any documentation to explain the JSON output? most of the fields are self-explanatory but not always

Appreciate for any help! 🙂

Below is what are in task_settings using an existing CDC task:

"task_settings": {

"source_settings": {

"target_settings": {

"queue_settings": {

"message_shape": {

"key_shape": {

}

"ftm_settings": {

}

"sorter_settings": {

"local_transactions_storage": {

}

"common_settings": {

"change_table_settings": {

"handle_ddl": false,

"header_columns_settings": {

}

"audit_table_settings": {

"apply_changes_enabled": false,

"dr_settings": {

"statistics_table_settings": {

"bidi_table_settings": {

"task_uuid": "123-abc",

"status_table_settings": {

"suspended_tables_table_settings": {

"history_table_settings": {

"exception_table_settings": {

"recovery_table_settings": {

"data_batching_settings": {

"data_batching_table_settings": {

"log_stream_settings_depricated": {

"ddl_history_table_settings": {

"customized_charset_settings": {

"validation": {

"sub_char": 0

}

Re: How do I know whether a task is full load or CDC using the metadata JSON?

Dana_Baldwin — Wed, 10 Jan 2024 22:45:14 GMT

Hi @S_Lo_DE

I'm afraid we don't have any documentation on the task JSON file. I can say that most of the default settings are not explicitly written to the JSON file when it's exported. Full load & apply changes are enabled by default, so they are not written to the JSON.

If you disable both of them and then enable store changes (at least one of these 3 must be enabled), you'll see this under the common_settings section:

"full_load_enabled": false,

"apply_changes_enabled": false,

"save_changes_enabled": true,

Hope this helps

Dana

Re: How do I know whether a task is full load or CDC using the metadata JSON?

Yadunandan — Thu, 11 Jan 2024 01:06:56 GMT

Hello @S_Lo_DE

If you have configured your task on default settings, that is Full load + CDC then the JASON will not contain the details you are looking for, if you have made some changes to it, that is Full load only or full load + store change or CDC only and like wise, you will have those details mentioned under the "common_settings section".

Regards,
Yadunandan

Re: How do I know whether a task is full load or CDC using the metadata JSON?

S_Lo_DE — Thu, 11 Jan 2024 18:42:46 GMT

Thanks @Dana_Baldwin and @Yadunandan , that is really helpful and seem there is the only way i can work out the task type 🙂

Can I please ask if there is a way I can know whether a CDC task is currently running or not in the JSON file? For a full load task, it's easy to just look at scheduler.

Re: How do I know whether a task is full load or CDC using the metadata JSON?

Dana_Baldwin — Thu, 11 Jan 2024 18:59:41 GMT

Hi @S_Lo_DE

The current task status is not written to the task JSON file as it is only the task definition. There are ways to tell if a task is running in CDC mode without viewing the user interface for each task.

Our Enterprise Manager product can show the status of multiple tasks on one screen. It also has API call functionality that can return a task's status. I'm not sure if you use this product but here's a link to the documentation: Start | Qlik Enterprise Manager Help

There are repctl commands that can be used (precursors to the API calls) but they are not documented.

Use this command to get the general instructions on how to use repctl:

C:\Program Files\Attunity\Replicate\bin>repctl help

To get the list of commands:

C:\Program Files\Attunity\Replicate\bin>repctl help ?

command help:

request arguments:

option:optional enum, valid values:

01 - HELP_COMMAND_LIST

02 - HELP_OBJECT_LIST

03 - HELP_ALL

command:optional string

error:optional int32

[help command] Succeeded

I would suggest using option #3 and redirecting to a file to get the complete list,

Like this:

C:\Program Files\Attunity\Replicate\bin>repctl help option=03 > C:\Temp\export.txt

Hope this helps.

Dana

Re: How do I know whether a task is full load or CDC using the metadata JSON?

Heinvandenheuvel — Thu, 11 Jan 2024 20:05:24 GMT

@Dana_Baldwin has some good info there. Read it, and the userguide which will explain how to use '-d' for a non-default data directory. Personnaly I prefer : repctl -d <mydata> help option=1 > commandlist.txt. The I scan the list for example with grep or on windows findstr 'status' to see what is there and finally repctl -d <mydata> gettaskstatus ? to see details for the command you most likely want. Actually, you are likely to end up with repctl -d <mydata> connect; getserverstatus The connect command will get you 'live' data and server status returns a handy JSON array of task/state/task_type.

But just to see if a task is running, you may also want to look Operating System commands like for Linux PS -EF | grep [r]epctl

For windows that could be cmd tasklist but that does not give command line info. I like Powershell better. Check out the following 'one-liner':

$tmp = $env:COMPUTERNAME + ' ' + (Get-Date -Format "yyyy-MM-dd HH:mm:ss") Get-WmiObject Win32_Process -filter "name like 'rep%.exe'" | Format-Table -AutoSize @{Label="Pid"; Expression={$_.ProcessId}}, @{ label=$tmp; expression={ (($_.CommandLine -replace("^.*?reptasksrv","")) + ' '*30 ).substring(1,30) }}, # @{ label=$tmp; expression={ (($_.CommandLine).substring(1,60) )}}, @{ Label="Elapsed(s)"; Expression={[math]::floor((New-TimeSpan $_.ConvertToDateTime($_.CreationDate)).TotalSeconds)}}, @{ Label="CPU(s)"; Expression={[math]::floor(($_.UserModeTime + $_.KernelModeTime) / 10000000)}}, @{ Label="Reads"; Expression={$_.ReadOperationCount}}, @{ Label="Writes"; Expression={$_.WriteOperationCount}}, @{ Label="VM(MB)"; Expression={[math]::floor($_.VM/1048576)}}, @{ Label="WS(MB)"; Expression={[math]::floor($_.WS/1048576)}}

Re: How do I know whether a task is full load or CDC using the metadata JSON?

S_Lo_DE — Thu, 18 Jan 2024 18:22:02 GMT

Thanks heaps @Dana_Baldwin , @Yadunandan and @Heinvandenheuvel , it helps a lot 🙂

I will see if I can use the enterprise manager api to get live data. My situation/environment is a bit tricky and I'm afraid that I(non-admin) won't be able to run/set up a job to run those commands to server

Re: How do I know whether a task is full load or CDC using the metadata JSON?

SushilKumar — Fri, 19 Jan 2024 01:27:49 GMT

Hello team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar