topic Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large in Talend Studio

tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Fri, 15 Nov 2024 23:07:40 GMT

Hi all,

I develop an ETL pipeline with TOS 8. Pipeline read from SQL Server and write to Google BigQuery with tBigQueryOutput but I received this error:

Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

at java.base/java.nio.file.Files.readAllBytes(Files.java:3212)

at lov_poc.job_skuitemtag_1_1.Job_SkuItemTag.tDBInput_1Process(Job_SkuItemTag.java:11010)

at lov_poc.job_skuitemtag_1_1.Job_SkuItemTag.tDBConnection_1Process(Job_SkuItemTag.java:1005)

at lov_poc.job_skuitemtag_1_1.Job_SkuItemTag$2.run(Job_SkuItemTag.java:21841)

Here the flow details:

The CSV is correctly written on the local file system (5GB) but is not copied to Google Cloud Storage.

The same process is correctly execute by TOS 7.3.

My JVM:

has anyone had the same problem?

Thank you,

Federico

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

gjeremy1617088143 — Fri, 11 Mar 2022 11:04:13 GMT

Hi, do you use a 32 bit jre or 64bit ?

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Fri, 11 Mar 2022 11:12:09 GMT

Hi @guenneguez jeremy

32 bit

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Fri, 11 Mar 2022 12:06:23 GMT

I tried with 64 jre:

java -version

openjdk version "11.0.14.1" 2022-02-08 LTS

OpenJDK Runtime Environment Zulu11.54+25-CA (build 11.0.14.1+1-LTS)

OpenJDK 64-Bit Server VM Zulu11.54+25-CA (build 11.0.14.1+1-LTS, mixed mode)

But I received the same error.

Thanks,

Federico

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

gjeremy1617088143 — Fri, 11 Mar 2022 13:12:50 GMT

I ask it cause the jre 32 bit could throw thos type of error

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Fri, 11 Mar 2022 13:26:19 GMT

ok. 😞

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

Anonymous — Mon, 14 Mar 2022 04:29:59 GMT

Hello,

No target table was created in BigQuery and no load was started from the GCS staging bucket?

Have you tried to add a few JVM parameters? Based on what has been mentioned so far, we would suggest to increase the amount of memory that is set for the job itself.

Please try increasing more value. please monitor the job while it running to check how many memory it really utilizing.

Hope this KB article helps

https://community.talend.com/s/article/OutOfMemory-Exception-WmtmQ

Best regards

Sabrina

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Mon, 14 Mar 2022 08:15:06 GMT

Hello,

"No target table was created in BigQuery and no load was started from the GCS staging bucket?"

Exactly. Only the csv on the local filesystem is created.

"Have you tried to add a few JVM parameters? Based on what has been mentioned so far, we would suggest to increase the amount of memory that is set for the job itself."

YES:

Also I checked the memory run tab with this parameters and no problem is detected.

I think the problem is due to the version of the libraries it uses.

Infact if I run the project build with TOS 7.3 and TOS 8 and compare them the latter in the .bat file uses newer libraries.

But I don't know how to solve this problem.

Thanks again,

Federico

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

Anonymous — Mon, 14 Mar 2022 09:13:26 GMT

Hello,

Increasing heap doesn't help?

Looks like it happens due to the max size limitation in java.

The same process is correctly execute by TOS 7.3 and what's JDK you are using in talend studio V 7.3.1? JDK 1.8 and 64 bit? Are you running on 64 bit OS as well? Did you get the same issue when use oracle JDK 11?

Best regards

Sabrina

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Mon, 14 Mar 2022 10:15:22 GMT

I use the same configuration in TOS 7.3 and TOS 8 and run under the same OS.

The OS is 64bit.

JDK installed is:

thanks again

Federico

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

gjeremy1617088143 — Mon, 14 Mar 2022 10:31:51 GMT

in the project properties, under Build --> java version wich one is selected ?

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Mon, 14 Mar 2022 10:46:59 GMT

TOS 7.3:

TOS 8:

I cannot change it in either version. In the drop-down menu there is only 1.8.

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

gjeremy1617088143 — Mon, 14 Mar 2022 10:50:21 GMT

and in the .ini file of you talend do you have specified a specific Java version ?

cause as i see it's seem you don't use jdk 11 in tos8

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Mon, 14 Mar 2022 10:59:30 GMT

are you talking about these files?

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

gjeremy1617088143 — Mon, 14 Mar 2022 12:09:42 GMT

at the top of your ini file you can put this :

-vm

C:\Program Files\Java\jdk11\bin\ (here your path to the jdk11 bin)

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Mon, 14 Mar 2022 12:49:24 GMT

I have tried and this attempt does not work either.

I have tried also this path: C:\Program Files\Zulu\zulu-11\bin

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

Anonymous — Tue, 15 Mar 2022 04:06:29 GMT

Hello,

Could you please open an issue on Talend Bugtracker and our developers from R&D will check your problem to see if it is an issue from libraries.

Best regards

Sabrina

Re: tBigQueryOutput Exception in thread "Thread-1" java.lang.OutOfMemoryError: Required array size too large

ffanali0804 — Tue, 15 Mar 2022 08:27:07 GMT

done: https://jira.talendforge.org/browse/TFD-14038

Thanks again

Federico