Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 
LvffY
Contributor II
Contributor II

Use kafka partitioned topics as target

Hello,

I would like to use a Full Load process with a MSSQL server as source adn a Kafka topic as target. 

I already have a working process. This process targets a "all-default" kafka topic. In particular, the topic has only one partition because it has been created automatically when first used. 

Now I would like to have more than one partition to my topic (to increase read & write speed) but when I try to write into my topics, I have a weird error "Failed to produce kafka message with record id <2> to partition <-1> in topic 'dbo.FinancialTest'. Local: Invalid argument or configuration [1020401] (queue_utils.c:104)". 

Do you have any idea about the problem ? 

Thanks in advance,

LvffY

P.S : I attach the full logs and the output of the export task for anyone who would want to have more informations

Labels (1)
1 Solution

Accepted Solutions
LvffY
Contributor II
Contributor II
Author

I found my mistake in adding the verbose logs. 

Actually this was an ACL error, I wasn't able to write into my target topic.

Thanks for your help anyway !

View solution in original post

3 Replies
john_wang
Support
Support

Hello @LvffY ,

May I know what's the Confluent version?

Looks to me the issue relevant to the partitioning stage. Try different partition numbers, use different message key to do partitioning (schema and table name, primary key columns, or none), and enable target_load to verbose then check task log files to see if helpful clue there. Or reproduction steps help to understand the scenario.

Hope this helps.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
LvffY
Contributor II
Contributor II
Author

Hello @john_wang 

We're using confluent cloud. I don't know if there is any limitations on the cloud version. 

I already tried multiple partition system but I'll try to add verbose logs. 

LvffY
Contributor II
Contributor II
Author

I found my mistake in adding the verbose logs. 

Actually this was an ACL error, I wasn't able to write into my target topic.

Thanks for your help anyway !