We have use case where we want to convert 17TB of data(csv files) into parquet. we are using EMR spark cluster for conversion. we have designed Big data job with tFileOutputParquet component to create the file. currently our job is taking long time to convert the files. did anyone achieved parquet conversion with alternate approach and optimized design? kindly share some inputs if known.
