idea Replicate task with Kafka target - memory usage in Suggest an Idea

Replicate task with Kafka target - memory usage

mgarret2 — Mon, 06 Jul 2020 13:43:01 GMT

Kafka tasks are consuming high memory and the memory is not being released unless we stop and resume the tasks. We have used the internal parameter trackerMaxMessages provided by support team, we saw an improvement but still we have a few tasks which are consuming high memory. We would like to see improvement on memory usage by these type of tasks.

Re: Replicate task with Kafka target - memory usage - Status changed to: Open - On Roadmap

Ola_Mayer — Mon, 13 Jul 2020 13:55:53 GMT

Re: Replicate task with Kafka target - memory usage

mgarret2 — Mon, 17 Aug 2020 18:01:14 GMT

Hi Ola,

We are starting to move more and more jobs to Kafka as a target. This lets us de-couple from a specific target like HDP/CDP/Teradata. I like this architecture and like that you have this item on the road map. May we know if you have any ETA on when we would see something?

Re: Replicate task with Kafka target - memory usage - Status changed to: Delivered

Shelley_Brennan — Wed, 12 May 2021 13:42:35 GMT

We do have improvements to memory consumption for Kafka targets in the May 2021 release, now available. Will be happy to take any feedback once you've had a chance to try it in your environment. Thank you!