ComposeFDL - CDC task fails on error: "Hash table loading exceeded memory limits for input"

[David_Fergen]

Hash table loading exceeded memory limits for input.

In Compose CDC task, the following error appears while running a merge query on a big table:

MapJoinMemoryExhaustionError: Hash table loading exceeded memory limits for input: Map 6 numEntries: 58900000 estimatedMemoryUsage: 4085283484 effectiveThreshold: 4081494022

Resolution: 

Keep table, partition and column statistics up-to-date for Hive query compiler to generate optimal query plan.
 
References
https://community.cloudera.com/t5/Community-Articles/Map-Join-Memory-Sizing-For-LLAP/ta-p/247462

CAUSE:

Memory monitoring during hashtable loading monitors hash table's in-memory size for fair usage. If memory monitor finds an executor using more memory beyonds its limits during hashtable loading, the query will fail with mapjoin memory exhaustion error.