topic Re: Use kafka partitioned topics as target in Qlik Replicate

Use kafka partitioned topics as target

LvffY — Tue, 15 Feb 2022 11:00:19 GMT

Hello,

I would like to use a Full Load process with a MSSQL server as source adn a Kafka topic as target.

I already have a working process. This process targets a "all-default" kafka topic. In particular, the topic has only one partition because it has been created automatically when first used.

Now I would like to have more than one partition to my topic (to increase read & write speed) but when I try to write into my topics, I have a weird error "Failed to produce kafka message with record id <2> to partition <-1> in topic 'dbo.FinancialTest'. Local: Invalid argument or configuration [1020401] (queue_utils.c:104)".

Do you have any idea about the problem ?

Thanks in advance,

LvffY

P.S : I attach the full logs and the output of the export task for anyone who would want to have more informations

Re: Use kafka partitioned topics as target

john_wang — Tue, 15 Feb 2022 14:43:23 GMT

Hello @LvffY ,

May I know what's the Confluent version?

Looks to me the issue relevant to the partitioning stage. Try different partition numbers, use different message key to do partitioning (schema and table name, primary key columns, or none), and enable target_load to verbose then check task log files to see if helpful clue there. Or reproduction steps help to understand the scenario.

Hope this helps.

Regards,

John.

Re: Use kafka partitioned topics as target

LvffY — Tue, 15 Feb 2022 15:26:31 GMT

Hello @john_wang

We're using confluent cloud. I don't know if there is any limitations on the cloud version.

I already tried multiple partition system but I'll try to add verbose logs.

Re: Use kafka partitioned topics as target

LvffY — Tue, 15 Feb 2022 16:17:46 GMT

I found my mistake in adding the verbose logs.

Actually this was an ACL error, I wasn't able to write into my target topic.

Thanks for your help anyway !