How to add a custom header to qlik replicate task

[RajneeshSharma]

I have a requirement where the customer is asking to add a custom header key value to a qlik replicate task that reads from a table and writes to kafka. The output is in JSON format. i tried several things like creating a new column in Global rules, using custom message format in Task Settings but it doesn't provide any feature to define your own headers. There are only a header/ section and some predefined headers like Operation, Change sequence, Tiemstamp, Stream position etc. 

Any idea how to implement custom headers. Thanks.

[Bill_Steinagle]

@RajneeshSharma 

Thank you for the post to the QDI Forums. If you need to set a new Column you can use the Add Column to the Target and set the Column Value per you requirements for the Column you define. You can use the Expression Builder on the Column you add. Here is the link to the User Guide for 2023.5 and can change the version as needed per this is web link. Also there are many links on the Community as well for Transformation and Expression Builder.

https://help.qlik.com/en-US/enterprise-manager/May2023/Content/Global_Common/Content/SharedEMReplica...

Expression Builder:

https://help.qlik.com/en-US/enterprise-manager/May2023/Content/Global_Common/Content/SharedEMReplica...

Expression builder uses the SQL Lite syntax when creating your expression for the new column.

https://www.sqlite.org/index.html

Regards,

Bill 

[RajneeshSharma]

Thanks Bill for responding. 

Once we have defined a custom column using Add column,  how can we set it up to go as a header in kafka JSON target ? Thanks

[Bill_Steinagle]

@Ran 

Hello thank you for the reply. As per the link below I would check the Limitations as the Target added column should give you more details on its usage. You can setup the Transform and do the Add Column option on the Targets to be generated in Kafka. On that column you can setup a Expression to add or how the data is landed or generated in this new Column. Hope this helps. If you need more in depth information you may share a new Support case as it also could be a new feature to the tool which would require a Ideation you can create from the Community. 

https://help.qlik.com/en-US/replicate/May2023/Content/Replicate/Main/Kafka/limitations_kafka.htm#Lim...

Thanks,

Bill

[RajneeshSharma]

To make it more clear, I want the yellow shaded key:value to come in headers[] array. Refer attached screenshot.

[john_wang]

Hello @RajneeshSharma ,

If I understand you correctly, you are trying to add some KV pairs (or special datas) into Kafka message HEADERS part rather than DATA part. is that correct?

In Qlik Replicate, we may add additional KV pairs into messages HEADERS part or DATA part depending on how the definition is. Please check the article: Qlik Replicate and Kafka target: How to add tables name in Kafka message.

Hope it helps.

Regards,

John.

[RajneeshSharma]

Thanks John for informative post. However, it still doesn't solve my requirement. Firstly, I am on Nov 2021 version and secondly my client is asking to put custom header in headers[] block ( which to my knowledge is a kafka provided header) and not header{} block ( refer screenshot attached)

[Heinvandenheuvel]

Well, that is not going to happen.

You'll have to inform your customer 'No can do'.

What are they trying to achieve? can a workaround - no matter how clumsy - be constructed?

Hein.