Merge multiple csv files each having different schema.

[Anonymous]

I have a job where multiple csv files are generated each having a different schema.I need to merge all the csv files into one csv file.Could anyone please guide me on achieving this.

[willm1]

Perfect example here... https://help.talend.com/search/all?query=tMap&content-lang=en
Scroll down to the Scenarios...

[Anonymous]

Hi saideshwarpalvai
All different schema must have at least some fields in common. Do you want these common fields as one output file or do you expect all existing fields in the output file with the backdraft of a lot of them are empty?
If your csv files have mostly a similar schema but in different field order you could use the tFileInputTextFlat
http://talendforge.org/exchange/index.php?eid=745&product=tos&action=view&nav=1,1,1
component to get the field extraction automatically configured by a header line (also possible with the help of regex).

[Anonymous]

Hi jlolling,
All the csv files would have different schemas and some of them would have blank values as well.I just need to append all the rows from the different csv files into one file.There would be some common fields in all the csv files but i have to consider all the fields apart from the common fields and append all the rows.Please let me know if this can be achieved by tFileInputTextFlat component.

[willm1]

Hi Sai - if your schemas are defined ahead of time and do not change, you can manually build and use your schemas... See http://www.talendforge.org/tutorials/tutorial.php?idTuto=5
But as Jan said above, you can use the component he built to create your schemas dynamically (it does a lot more things!) at run time. Remember you have to include that component in your Studio to use it...

[Anonymous]

Hi willlm,
Thanks for providing the information.
I have attached the screenshot of the job which i have developed.The tXmlMap component outputs three csv files and each of the three csv files have different number of columns.Now the next step in the job is to merge these csv files with different number of columns.I have yet to try with the component which jan mentioned.Apart from that do we have any other possibility to perform the merge operation.

[Anonymous]

Try using the tFileOutputMSDelimited. This will allow you to write multiple different schema to the same delimited file.