topic Re: Kafka attrep_apply_exceptions topic in Qlik Replicate

Kafka attrep_apply_exceptions topic

BradA — Tue, 25 Aug 2020 21:29:27 GMT

I am using Replicate 6.5.0.293. When using a Kafka target configured with Avro format and Confluent Schema Registry across multiple tasks, I'd like to be able to manage the message format/schema compatibility (using schema registry) for the control table/topic separately from the replication target topics. With multiple tasks where different message formats are applied, compatibility issues associated with the schema applied to "attrep_apply_exceptions" can create errors at the task level.

My current method for dealing with this is to override the topic name for attrep_apply_exceptions using the task...task_settings...common_settings..exception_table_settings value in the task JSON file allowing a separate exceptions topic for each Task. My desire would be to leave the message format of the exceptions topic simple (no modifications) and turn off schema compatibility checking for it. This would avoid compatibility issues that may be created when message formats are different or evolve across tasks that use the same Kafka target and exceptions topic. This would also mitigate the need to consider using a different exception topic per task to avoid compatibility issues across tasks.

Is there a way to turn off schema compatibility checking for the exceptions table/topic only?

Re: Kafka attrep_apply_exceptions topic

David_Fergen — Thu, 01 Oct 2020 19:56:36 GMT

Hi BradA,

I will try to spread some light on what you can do for your request. So currently, there is no way in 6.5 but here is the workaround.

Add Schema for Control tables per task.

Also, In 6.6, we improved this:

For customers who use Confluent and Hortonworks Schema Registry for Kafka endpoint, Replicate will create the control table subjects in No Compatibility mode (this mode can be modified using an internal parameter setNonCompatibilityForControlTables. Default is true).
A special Control Tables Namespace, which excludes the Task name and Schema name, will be defined (this mode can be changed in the Task setting Import/Export using an internal parameter control_table_namespace)

Also, if you have any features or ideas you want to submit, feel free to enter them in our ideation section!

https://community.qlik.com/t5/Ideas/idb-p/qlik-ideas

Hopefully, that helps!

Thanks,

David

Re: Kafka attrep_apply_exceptions topic

BradA — Mon, 23 Nov 2020 14:52:55 GMT

Hi David,

Can you give a little more details on how the control table namespace parameter is to be set? I am interpreting your explanation as to mean there is a manual setting that can be created in the task JSON files using the import/export. Can you show and example of where in the JSON the "control_table_namespace" setting goes?

Thanks,
Brad