Qlik Replicate- How to create Kafka control topics per task?

[adbdkb]

I have a task that would get the data from DB2 and write to Kafka.  I set-up the control topics following this doc - https://community.qlik.com/t5/Knowledge/How-to-use-control-tables-topics-attrep-status-attrep-histor... 

I have a couple of questions -

1. Can the control topic be set per task or for all tasks that have Kafka as target endpoint, the the tables will be common
2. Is there a way to set these up such that the topic gets updated only if there is an update when using incremental.  It seems, there is a message in the topic every few seconds even if there is no data coming in from source endpoint.
3. If there are multiple tables in a Task, is there a way to get the separate count for each of the table, for updates, instead of single value that has count of all updates in all tables?

Thank you

[SwathiPulagam · Accepted solution]

Hi @adbdkb ,

For your 1st and 2nd questions below is the answer:

Yes we can have one control topic per task but you have to manually update the json file.
Example for one control topic.

You can rename this by exporting your task json and edit the following lines, and import back:

"exception_table_settings": {

"table_name": "attrep_apply_exceptions_temp"

},

This will rename the table to attrep_apply_exceptions_temp

If this is what you expect for other control tables then please create a case. I will work with you.

Thanks,

Swathi

[adbdkb]

Hi @SwathiPulagam ,

I exported the task.  This is what that part looks like.  What you are saying is that in the settings that are enabled - basically like defining the topic name that is enabled?  

"status_table_enabled": true, - I should define it like 

"status_table_settings": {},

as

"status_table_settings": {
      "table_name": "attrep_status_mytaskname_topic"
},

And create those topics ahead of time?

"status_table_settings": {},
"status_table_enabled": true,
"suspended_tables_table_settings": {},
"suspended_tables_table_enabled": true,
"history_table_settings": {},
"history_table_enabled": true,
"exception_table_settings": {},
"recovery_table_settings": {},
"data_batching_settings": {},
"data_batching_table_settings": {},
"log_stream_settings_depricated": {},
"ddl_history_table_settings": {},

 
This would take care of question #1, right?

How do we set it up for question #2

Is there a way to set these up such that the topic gets updated only if there is an update when using incremental.  It seems, there is a message in the topic every few seconds even if there is no data coming in from source endpoint.

Thanks

[adbdkb]

Trying to open a new question, but keep on getting authentication error.  So, checking if I can reply to an existing topic. ( Happening even after I tried logging off and then on )

[SwathiPulagam · Accepted solution]

Hi @adbdkb ,

Yes, you are correct. The process you explained will cover your 1st question and you have to create a control topic before you start the task or if you have a auto create topic permission that will take care.

Thanks,

Swathi

[SwathiPulagam]

Hi @adbdkb ,

Here is my understanding of your 2nd question. Do you want to control the status table updates?

Thanks,

Swathi

[adbdkb]

Yes, that way the topic doesn't get flooded.  I think otherwise there is a message in the topic every few seconds - at least that is what it appears to be

[SwathiPulagam · Accepted solution]

Hi @adbdkb ,

By default, the updates made to the attrep_status table is 1 minute.

We can change the frequency of updates made to the attrep_status table from 1 minute to X minutes by adding a field in the json under sorter settings. You can refer to the below article:

https://community.qlik.com/t5/Knowledge/Qlik-Replicate-Process-to-change-the-frequency-of-updates-ma...

Thanks,

Swathi

[adbdkb]

Thanks @SwathiPulagam .  

What is the maximum value it can be set to ( How many minutes ? ) Also, what does "memory_keep_time" do?  What values should it be set for?

"cdc_notification_period":   600,
    "local_transactions_storage": {
             "memory_keep_time":  120
                      }

[adbdkb]

@SwathiPulagam  -  What are the considerations when using custom topic names in the json?

This is what I have but I get an error when importing the task.

"status_table_settings": {
  "table_name": "attrep_status_adb_sysname_tblnm_task1",
},
"status_table_enabled": true,
"suspended_tables_table_settings": {
  "table_name": "attrep_suspended_tables_adb_sysname_tblnm_task1"
},
"suspended_tables_table_enabled": true,
"history_table_settings": {
  "table_name": "attrep_history_adb_sysname_tblnm_task1"
},
"history_table_enabled": true,
"exception_table_settings": {
  "table_name": "attrep_apply_exceptions_adb_sysname_tblnm_task1"
},

The error I get is

SYS-E-HTTPFAIL, Cannot import repository from the file /opt/attunity/replicate/data/tmp/220711154636.

If I remove the custom topic names, I am able to import the task.

So, wondering, what are the rules that need to be followed for these topic names, like,

1.  Must they start with attrep_
2.  Must the name be all lower case
3.  Is there a limit on the length of the topic name

If you can help with identifying the case of the issue and provide some guidance on the names, that will be useful 

Thanks