Hi,
I have issue about converting csv to parquet with tFileOutputParquet.
But the component required winutils and c++ 2010 for windows.
After added HADOOP_HOME and run c++ file, component is creating empty .parquet file with error.
How can I resolve this issue?
Error output:
Starting job csv2parquet at 14:16 12/02/2024.
[statistics] connecting to socket on port 3431
[statistics] connected
[WARN ] 14:16:08 org.apache.hadoop.util.NativeCodeLoader- Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in component tFileOutputParquet_1 (csv2parquet)
ExitCodeException exitCode=-1073741515:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1007)
at org.apache.hadoop.util.Shell.run(Shell.java:900)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:867)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:254)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:234)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:333)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:322)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:353)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:403)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:466)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:445)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1105)
at org.apache.parquet.hadoop.util.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:81)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:246)
at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:280)
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:530)
at bidemo.csv2parquet_0_1.csv2parquet.tFileInputDelimited_1Process(csv2parquet.java:756)
at bidemo.csv2parquet_0_1.csv2parquet.runJobInTOS(csv2parquet.java:1689)
at bidemo.csv2parquet_0_1.csv2parquet.main(csv2parquet.java:1387)
[FATAL] 14:16:08 bidemo.csv2parquet_0_1.csv2parquet- tFileOutputParquet_1
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1007) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.util.Shell.run(Shell.java:900) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:867) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:254) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:234) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:333) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:322) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:353) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:403) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:466) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:445) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1125) ~[hadoop-common-3.2.4.jar:?]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1105) ~[hadoop-common-3.2.4.jar:?]
at org.apache.parquet.hadoop.util.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:81) ~[parquet-hadoop-1.10.1.jar:1.10.1]
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:246) ~[parquet-hadoop-1.10.1.jar:1.10.1]
at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:280) ~[parquet-hadoop-1.10.1.jar:1.10.1]
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:530) ~[parquet-hadoop-1.10.1.jar:1.10.1]
at bidemo.csv2parquet_0_1.csv2parquet.tFileInputDelimited_1Process(csv2parquet.java:756) [classes/:?]
at bidemo.csv2parquet_0_1.csv2parquet.runJobInTOS(csv2parquet.java:1689) [classes/:?]
at bidemo.csv2parquet_0_1.csv2parquet.main(csv2parquet.java:1387) [classes/:?]
[statistics] disconnected
Job csv2parquet ended at 14:16 12/02/2024. [Exit code = 1]