Not able to write data in text and parquet format simultaneously
Hi,
Here i have a big data job, i wanted to write the output of the job in to two files,
1.tFileoutputdelimited and
2.tFileoutputParquet.
I cannot accomplish this , but i could write in to an output if the other was disabled.
I could write in to tFileoutputParquet if tFileoutputdelimited was deactivated and vice versa.
And the error log is as follows, when i try to execute the job with both the output formats.
Exception;tRunJob_2;java.lang.RuntimeException:Child job returns 1. It doesn't terminate normally. Error: org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync! at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210) at etl_3_1.customerdataloadmr_0_1.CustomerDataLoadmr$olap_customerStructInputFormat$AvroRecordReader.next(CustomerDataLoadmr.java:64215) at etl_3_1.customerdataloadmr_0_1.CustomerDataLoadmr$olap_customerStructInputFormat$AvroRecordReader.next(CustomerDataLoadmr.java:1) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:203) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:189) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:458) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.io.IOException: Invalid sync! at org.apache.avro.file.DataFileStream.nextRawBlock(DataFileStream.java:293) at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:198) ... 12 more