Kafka tasks are consuming high memory and the memory is not being released unless we stop and resume the tasks. We have used the internal parameter trackerMaxMessages provided by support team, we saw an improvement but still we have a few tasks which are consuming high memory. We would like to see improvement on memory usage by these type of tasks.
We are starting to move more and more jobs to Kafka as a target. This lets us de-couple from a specific target like HDP/CDP/Teradata. I like this architecture and like that you have this item on the road map. May we know if you have any ETA on when we would see something?
We do have improvements to memory consumption for Kafka targets in the May 2021 release, now available. Will be happy to take any feedback once you've had a chance to try it in your environment. Thank you!
