topic Re: tsqoopimport : how to merge generated files in Talend Studio

tsqoopimport : how to merge generated files

Anonymous — Sat, 16 Nov 2024 08:47:17 GMT

hello,

I use the TSqoopimport component for importing oracle tables to HDFS.

This component generate 4 files (part-m-xxxx) if i configure it with 4 mappers.

and after, how can i merge thos 4 files into one file ?

I use TOS for Big Data 6.1.1

Re: tsqoopimport : how to merge generated files

Anonymous — Wed, 31 Jan 2018 07:02:46 GMT

Hello,

The source data comes from 2 different sources, but has the same schema?

You can use a tFileList to iterate on a tFileInput* row linked to a tFileOutputDelimited in append mode.

Let us know if it works.

Best regards

Sabrina

Re: tsqoopimport : how to merge generated files

Anonymous — Sat, 03 Feb 2018 22:31:55 GMT

If the data isn't huge, you can try configuring Sqoop to just use one mapper: that way, it will generate one file.

If you don't want to go that route, Sabrina is mostly correct, except that you'll need tHDFSFileList to iterate over the files. Instead of merging them, this will iterate over them, so you can do whatever ETL work you need to do.

David

Re: tsqoopimport : how to merge generated files

Anonymous — Wed, 07 Feb 2018 14:05:09 GMT

Ok, i've found : the thdfscopy as an option which can merge the files.