topic To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output. in Talend Studio

To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output.

Anonymous — Sat, 16 Nov 2024 05:58:30 GMT

I have different formated files in Source Directory - Eg: *.csv, *.dat,*.xls. I need to process each file from directory individually and process it to a single file- *.csv and move to hadoop directory.

Example : Let as assume we have 3 files- Pay_Slip_TV*.csv,Pay_Slip_Dish*.xls and Pay_Slip_Phone*.csv.

I need to pick each file individually from Source folder and process each file and combine together to one single - eg: Pay_Slip_Total.csv and copy to Hadoop directory.

Can anyone suggest method to do this process?

Re: To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output.

akumar2301 — Mon, 29 Apr 2019 13:27:29 GMT

This could be useful

https://community.talend.com/t5/Design-and-Development/merge-data-from-multiple-files-into-one/td-p/101893

You need to create separate subjob for XLS and CSV. but output can be same file in (Append mode)

Re: To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output.

Anonymous — Mon, 29 Apr 2019 14:50:48 GMT

@uganesh Thanks for the reply,

The output needs to be in same file, and that can be done by append mode.But while creating sub-job for XLS and CSV separately,how can we take each file individually and iterate through each sub-job. Do we need any java code (tJava)or can we do it with any other component in Talend?

tFileList -> tMap/tJava (for filtering the .csv or .xls files) -> Sub-Job ( for processing the .csv or .xls files) -> tMap (to do appending)--> tFileOutputDelimited (output)

Please correct me if I am wrong.

Re: To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output.

akumar2301 — Mon, 29 Apr 2019 15:13:45 GMT

To do this

tFileList ------iterator-------- Tjava ( does nothing )

---RunIf 1 with below condition ----------- your CSV subjob

((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION")).equalsIgnoreCase("csv")

---RunIf 2 with below condition ----------- your XLS subjob

((String)globalMap.get("tFileList_1_CURRENT_FILEEXTENSION")).equalsIgnoreCase("xls")

Change your condition according to your need.

Re: To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output.

Anonymous — Tue, 30 Apr 2019 13:16:27 GMT

Thanks for correcting me

This is working fine 🙂

Re: To process different File formats (.csv, .xls etc) as Source Input from a directory and to process in to single .csv file as Output.

akumar2301 — Tue, 30 Apr 2019 13:20:15 GMT

Thanks Nitin,

If all ok , mark the post as resolved. It helps others in community to find correct solution to their problem.