Kafka Error 'Local: Queue full'

[Dana_Baldwin]

A task using Kafka target gets the following error stack in the task log:

[TARGET_APPLY   ]E: Failed to produce kafka message with record id <12345> to partition <-1> in topic 'my-topic'. Local: Queue full [1020414] (queue_utils.c:102)

00002796: 2022-01-16T03:09:55 [TARGET_APPLY   ]E: Failed to produce message. [1020414] (queue_apply.c:136)

00002796: 2022-01-16T03:09:55 [TARGET_APPLY   ]E: Error executing data handler [1020414] (streamcomponent.c:1981)

00002796: 2022-01-16T03:09:55 [TASK_MANAGER   ]E: Stream component failed at subtask 0, component st_0_PROD_TARGET_KAFKA [1020414] (subtask.c:1368)

00002796: 2022-01-16T03:09:55 [TARGET_APPLY   ]E: Stream component 'st_0_PROD_TARGET_KAFKA' terminated [1020414] (subtask.c:1537)

Environment

• Replicate 6.6

Resolution

Please add to your rdkafka properties settings: queue.buffering.max.messages=200000 and if it is still not working, please increase this to 300k and then 500k. The default value for this setting is 100000.

If you continue to have an issue, please open a support case.