topic Re: tFileOutputParquet - creating Parquet files in Talend Studio

tFileOutputParquet - creating Parquet files

RMotta2408 — Fri, 15 Nov 2024 21:50:08 GMT

Hi there,

I created this very simple Job:

There's no problem when I run it in Talend Cloud Big Data.

But when I run the same Job on Talend Cloud Data Fabric, on the same computer, it gives me the followinh error:

"Starting job ReadParquetFile at 12:41 09/05/2023.

Error: Could not find or load main class Dhadoop.home.dir=C:\hadoop-3.3.0\bin

Caused by: java.lang.ClassNotFoundException: Dhadoop.home.dir=C:\hadoop-3.3.0\bin

Job ReadParquetFile ended at 12:41 09/05/2023. [Exit code = 1]".

Can anyone please help me solve this issue?

I appreciate it.

Rui

Anonymous — Wed, 10 May 2023 01:45:24 GMT

Hello @Rui Motta ,

It's a well known issue about hadoop on windows

see https://cwiki.apache.org/confluence/display/HADOOP2/WindowsProblems

The solution is:

git clone the repository https://github.com/steveloughran/winutils to your local machine
Set the environment variable %HADOOP_HOME% to point to the directory above the BIN dir containing WINUTILS.EXE, it need to restart the studio make it take into account.

setup a jvm parameter -Dhadoop.home.dir in the job's Run->Advanced settings page as the below

RMotta2408 — Thu, 11 May 2023 13:12:45 GMT

@Aiming Chen , thank you so much for your help.

It worked like it was supposed to.

I really appreciate all the help you've been giving.

Rui