When using tFileInputRaw to load file contents as a string, the Job fails with an OutOfMemoryError if the file size exceeds 2GB. This happens even if there is enough memory and the JVM parameter "-Xmx" value is increased to a higher value.
Cause
A Java String internally uses a char array (example: char[]) and the indices of an array is an integer. The maximum value of an integer is Integer.MAX_VALUE, which is 2^31 – 1 (or approximately 2 billion). So, you can store a file up to 2 GB in size as a String, and for which you need at least 4 GB memory to store as each char is 2 bytes in Java, plus additional ~4 GB memory for creating the String object, so, in total around 8GB of heap space.
Reading ~2 GB file into memory is not a good design. Avoid loading ~2GB large files into strings at once, split the file into smaller files, and then use tFileList to iterate over each file.
