Hi there, looking for help ASAP. I'm getting this error when trying to dedup a HUGE text pipe delimited file. It is about 120GB. When I get this error, I usually play with the JVM arguments and I get it to run, but in this case I am not able to remedy it so far with changing the JVM arguments. It processes to about the 50M row level and it has over 100 million rows.
The latest arguments I had were this:
https://www.screencast.com/t/Sr0GU4B8kq9
And this is my workflow which is pretty simple. I just need to dedup the GetCurrentFile.
https://www.screencast.com/t/YZ7wYsI8BXJ
Any thoughts?
Thank you
