tsqoopimport : how to merge generated files

[Anonymous]

hello,

I use the TSqoopimport component for importing  oracle tables to HDFS.

This component generate 4 files (part-m-xxxx) if i configure it with 4 mappers.

and after, how can i merge thos 4 files into one file ?

I use TOS for Big Data 6.1.1

[Anonymous]

Hello,

The source data comes from 2 different sources, but has the same schema?

You can use a tFileList to iterate on a tFileInput* row linked to a tFileOutputDelimited in append mode.

Let us know if it works.

Best regards

Sabrina

[Anonymous]

If the data isn't huge, you can try configuring Sqoop to just use one mapper: that way, it will generate one file.

If you don't want to go that route, Sabrina is mostly correct, except that you'll need tHDFSFileList to iterate over the files. Instead of merging them, this will iterate over them, so you can do whatever ETL work you need to do.

David

[Anonymous · Accepted solution]

Ok, i've found : the thdfscopy as an option which can merge the files.