topic Re: How to Read Avro/Parquet Files in AWS S3 in Talend Studio

How to Read Avro/Parquet Files in AWS S3

Bruce_Perez — Fri, 15 Nov 2024 21:50:16 GMT

I am trying to move data from AWS S3 with flat files(avro or parquet) in Talend. What component shall I use to read/extract it? Im using Talend DI v8. I've tried tFileInputDelimited but only reads csv or txt. Not sure if Im configuring it correctly.

Re: How to Read Avro/Parquet Files in AWS S3

Anonymous — Tue, 09 May 2023 08:46:42 GMT

Hello,

You can use tAvroInput component to read Avro format file, see https://help.talend.com/r/en-US/8.0/avro/tavroinput

use tFileInputParquet component to read Parquet format file, see https://help.talend.com/r/en-US/8.0/parquet/tfileinputparquet

if you can't find the components in studio, please install them by feature manager

see https://help.talend.com/r/en-US/8.0/studio-user-guide/install-features-to-talend-studio

Best regards

Aiming

Re: How to Read Avro/Parquet Files in AWS S3

Bruce_Perez — Tue, 09 May 2023 14:44:31 GMT

Thank you very much for your insights. Follow up question, is this feature available only in enterprise version, or can I install in free version?

Re: How to Read Avro/Parquet Files in AWS S3

Anonymous — Wed, 10 May 2023 01:04:14 GMT

hello @Bruce Perez ,

Unfortunately, this feature is only available in enterprise version. thanks

Re: How to Read Avro/Parquet Files in AWS S3

Bruce_Perez — Wed, 10 May 2023 07:19:30 GMT

Are there any alterntative component which I can use to read these files? For Avro, I have tried using tFileInputJSON, not sure with Parquet files. Again, I really appreciate your help on this.

Re: How to Read Avro/Parquet Files in AWS S3

rhall1 — Wed, 10 May 2023 09:03:56 GMT

You *may* be able to extrapolate from a blog a wrote a long time ago. Unfortunately it is no longer "live" but you can see it with "Wayback Machine". Here is a link....

http://web.archive.org/web/20200919171507/https://www.talend.com/blog/2019/06/12/talend-pipeline-designer-avro-schemas/

I put some code together to serialise and deserialise JSON data to and from AVRO. As I said, it is not exactly what you need, but you may be able to find a solution from this.