Kafka Error 'Local: Queue full'

Dana_Baldwin — Fri, 11 Feb 2022 23:51:43 GMT

A task using Kafka target gets the following error stack in the task log:

[TARGET_APPLY ]E: Failed to produce kafka message with record id <12345> to partition <-1> in topic 'my-topic'. Local: Queue full [1020414] (queue_utils.c:102)

00002796: 2022-01-16T03:09:55 [TARGET_APPLY ]E: Failed to produce message. [1020414] (queue_apply.c:136)

00002796: 2022-01-16T03:09:55 [TARGET_APPLY ]E: Error executing data handler [1020414] (streamcomponent.c:1981)

00002796: 2022-01-16T03:09:55 [TASK_MANAGER ]E: Stream component failed at subtask 0, component st_0_PROD_TARGET_KAFKA [1020414] (subtask.c:1368)

00002796: 2022-01-16T03:09:55 [TARGET_APPLY ]E: Stream component 'st_0_PROD_TARGET_KAFKA' terminated [1020414] (subtask.c:1537)

Environment

Replicate 6.6

Resolution

Please add to your rdkafka properties settings: queue.buffering.max.messages=200000 and if it is still not working, please increase this to 300k and then 500k. The default value for this setting is 100000.

If you continue to have an issue, please open a support case.

article Kafka Error 'Local: Queue full' in Official Support Articles

Kafka Error 'Local: Queue full'

Environment

Resolution