topic Re: Control topics - How to find actual updates in Qlik Replicate

Qlik Replicate- How to create Kafka control topics per task?

adbdkb — Thu, 22 Sep 2022 02:48:03 GMT

I have a task that would get the data from DB2 and write to Kafka. I set-up the control topics following this doc - https://community.qlik.com/t5/Knowledge/How-to-use-control-tables-topics-attrep-status-attrep-history/ta-p/1892522

I have a couple of questions -

Can the control topic be set per task or for all tasks that have Kafka as target endpoint, the the tables will be common
Is there a way to set these up such that the topic gets updated only if there is an update when using incremental. It seems, there is a message in the topic every few seconds even if there is no data coming in from source endpoint.
If there are multiple tables in a Task, is there a way to get the separate count for each of the table, for updates, instead of single value that has count of all updates in all tables?

Thank you

Re: Control topics - How to find actual updates

SwathiPulagam — Thu, 07 Jul 2022 20:50:01 GMT

Hi @adbdkb ,

For your 1st and 2nd questions below is the answer:

Yes we can have one control topic per task but you have to manually update the json file.
Example for one control topic.

You can rename this by exporting your task json and edit the following lines, and import back:

"exception_table_settings": {

"table_name": "attrep_apply_exceptions_temp"

This will rename the table to attrep_apply_exceptions_temp

If this is what you expect for other control tables then please create a case. I will work with you.

Thanks,

Swathi

Re: Control topics - How to find actual updates

adbdkb — Thu, 07 Jul 2022 23:38:04 GMT

Hi @SwathiPulagam ,

I exported the task. This is what that part looks like. What you are saying is that in the settings that are enabled - basically like defining the topic name that is enabled?

"status_table_enabled": true, - I should define it like

"status_table_settings": {},

"status_table_settings": {
"table_name": "attrep_status_mytaskname_topic"
},

And create those topics ahead of time?

"status_table_settings": {},
"status_table_enabled": true,
"suspended_tables_table_settings": {},
"suspended_tables_table_enabled": true,
"history_table_settings": {},
"history_table_enabled": true,
"exception_table_settings": {},
"recovery_table_settings": {},
"data_batching_settings": {},
"data_batching_table_settings": {},
"log_stream_settings_depricated": {},
"ddl_history_table_settings": {},

This would take care of question #1, right?

How do we set it up for question #2

Is there a way to set these up such that the topic gets updated only if there is an update when using incremental. It seems, there is a message in the topic every few seconds even if there is no data coming in from source endpoint.

Thanks

Re: Control topics - How to find actual updates

adbdkb — Fri, 08 Jul 2022 15:55:19 GMT

Trying to open a new question, but keep on getting authentication error. So, checking if I can reply to an existing topic. ( Happening even after I tried logging off and then on )

Re: Control topics - How to find actual updates

SwathiPulagam — Fri, 08 Jul 2022 17:01:18 GMT

Hi @adbdkb ,

Yes, you are correct. The process you explained will cover your 1st question and you have to create a control topic before you start the task or if you have a auto create topic permission that will take care.

Thanks,

Swathi

Re: Control topics - How to find actual updates

SwathiPulagam — Fri, 08 Jul 2022 17:02:30 GMT

Hi @adbdkb ,

Here is my understanding of your 2nd question. Do you want to control the status table updates?

Thanks,

Swathi

Re: Control topics - How to find actual updates

adbdkb — Fri, 08 Jul 2022 17:29:42 GMT

Yes, that way the topic doesn't get flooded. I think otherwise there is a message in the topic every few seconds - at least that is what it appears to be

Re: Control topics - How to find actual updates

SwathiPulagam — Fri, 08 Jul 2022 19:34:30 GMT

Hi @adbdkb ,

By default, the updates made to the attrep_status table is 1 minute.

We can change the frequency of updates made to the attrep_status table from 1 minute to X minutes by adding a field in the json under sorter settings. You can refer to the below article:

https://community.qlik.com/t5/Knowledge/Qlik-Replicate-Process-to-change-the-frequency-of-updates-made/ta-p/1943586

Thanks,

Swathi

Re: Control topics - How to find actual updates

adbdkb — Fri, 08 Jul 2022 19:42:46 GMT

Thanks @SwathiPulagam .

What is the maximum value it can be set to ( How many minutes ? ) Also, what does "memory_keep_time" do? What values should it be set for?

"cdc_notification_period":   600,
    "local_transactions_storage": {
             "memory_keep_time":  120
                      }

Re: Control topics - How to find actual updates

adbdkb — Mon, 11 Jul 2022 19:50:47 GMT

@SwathiPulagam - What are the considerations when using custom topic names in the json?

This is what I have but I get an error when importing the task.

"status_table_settings": {
  "table_name": "attrep_status_adb_sysname_tblnm_task1",
},
"status_table_enabled": true,
"suspended_tables_table_settings": {
  "table_name": "attrep_suspended_tables_adb_sysname_tblnm_task1"
},
"suspended_tables_table_enabled": true,
"history_table_settings": {
  "table_name": "attrep_history_adb_sysname_tblnm_task1"
},
"history_table_enabled": true,
"exception_table_settings": {
  "table_name": "attrep_apply_exceptions_adb_sysname_tblnm_task1"
},

The error I get is

SYS-E-HTTPFAIL, Cannot import repository from the file /opt/attunity/replicate/data/tmp/220711154636.

If I remove the custom topic names, I am able to import the task.

So, wondering, what are the rules that need to be followed for these topic names, like,

1. Must they start with attrep_
2. Must the name be all lower case
3. Is there a limit on the length of the topic name

If you can help with identifying the case of the issue and provide some guidance on the names, that will be useful

Thanks

Re: Control topics - How to find actual updates

SwathiPulagam — Tue, 12 Jul 2022 13:23:03 GMT

@adbdkb you left a comma after the status table name. Please remove the below comma which I highlighted.

 "table_name": "attrep_status_adb_sysname_tblnm_task1",

Thanks,

Swathi

Re: Control topics - How to find actual updates

adbdkb — Tue, 12 Jul 2022 13:33:49 GMT

Thanks @SwathiPulagam - that worked. Really appreciate your help.

What is the maximum value it can be set to ( How many minutes ? ) I am trying to set it to 21600 ( 6 hrs ). Is that valid?

Also, what does "memory_keep_time" do? What values should it be set for?

"cdc_notification_period":   600,
    "local_transactions_storage": {
             "memory_keep_time":  120
                      }

Is there any place where I can check what the json attributes are for and their possible values / ranges, if any?

Thanks again for your support @SwathiPulagam

Re: Control topics - How to find actual updates

SwathiPulagam — Thu, 14 Jul 2022 14:02:57 GMT

Hi @adbdkb ,

These are undocumented parameters, support will provide these parameters based on the requirement.

As i am aware, no minimum and maximum values are defined. you can test for 6 hrs and create a support case if you encounter any issues.

Thanks,

Swathi

Re: Control topics - How to find actual updates

SwathiPulagam — Thu, 14 Jul 2022 14:13:36 GMT

Hi @adbdkb ,

"local_transactions_storage": {

"memory_keep_time": 120

}

The above setting will represent Transactional offload tuning. You can leave the default value as it is 60 seconds. These values are tuned by the Support or Professional Services team when required.

Thanks,

Swathi