idea Re: Support Avro and Parquet file format in File target endpoint in Replicate - Status changed to: Open - Collecting Feedback in Suggest an Idea

Support Avro and Parquet file format in File target endpoint in Replicate

Prabodh — Wed, 25 Nov 2020 20:53:42 GMT

We would like to use File target endpoint to populate our datalake using Replicate. However, our datalake file format is Avro which is not currently supported by Replicate.

Please add support for Avro and Parquet file formats in File target endpoint.

Re: Support Avro and Parquet file format in File target endpoint in Replicate

simonaubert — Thu, 26 Nov 2020 15:32:50 GMT

Hello@Prabodh

May be a stupid question, but why don't you use Orc instead of Avro?

It's fastest and has better compression. Also, it works great with hive or druid.

Best regards,

Simon

Re: Support Avro and Parquet file format in File target endpoint in Replicate - Status changed to: Open - Collecting Feedback

Shelley_Brennan — Mon, 28 Dec 2020 14:36:53 GMT

Thank you for the feedback. Do you have any more details on your use case and if you've considered using Qlik Compose to build out the data lake, which does support Avro and Parquet file formats?

Re: Support Avro and Parquet file format in File target endpoint in Replicate

Prabodh — Mon, 15 Mar 2021 14:35:30 GMT

@simonaubert Avro is our standard data lake format.

@Shelley_Brennan Compose would not work for us. We need Replicate to land files in Avro format. Also, to be specific, we need the capability to write Avro files for S3 endpoint on priority.

Re: Support Avro and Parquet file format in File target endpoint in Replicate - Status changed to: Open - On Roadmap

Shelley_Brennan — Fri, 23 Jul 2021 18:09:46 GMT

Support for additional file formats is in our roadmap. We will update when development is underway. Please continue to share any additional feedback here in the meantime. Thank you!

Re: Support Avro and Parquet file format in File target endpoint in Replicate - Status changed to: Partially Delivered

Nulee_Massaro — Fri, 03 Jun 2022 18:01:19 GMT

Partially delivered - The option to create files in Parquet format has been added to the Amazon S3 target endpoint.