TIS : append data into parquet file

Anonymous — Sat, 16 Nov 2024 10:15:49 GMT

Hi,
I am using TIS Version: 6.2.1 .

I want to read data from Teradata and write it into HDFS (HDP v2.4, scoop > 1.4.6) with parquet file format
My first thought was to use tScoopImport component
The problem is that the version of sqoop deployed in my cluster is 1.4.6 which is buged and it doesn't support custom sql.
So I turned to parquet file :
I created a spark job and I used tFileOutputParquet that works juste fine but he didn't support appending data
to the same file even when I do set column partition !
So my question is : How can I append data in a parquet file written in HDFS ?
Thanks you a lot dor your help.

Re: TIS : append data into parquet file

Anonymous — Mon, 21 Nov 2016 09:21:54 GMT

Hi,
Per TalendHelpCenter:Which big data formats are supported, p arquet is not suppored on tSqoopImport
Could you please take a look at this jira issue: https://jira.talendforge.org/browse/TBD-3392 to see if the workaround is Ok with you?
Best regards
Sabrina

topic Re: TIS : append data into parquet file in Talend Studio

TIS : append data into parquet file

Re: TIS : append data into parquet file