topic Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table in Talend Studio

Loading multiple CSV files to one CSV file and then to Datawarehouse table

xx_emanis — Sat, 16 Nov 2024 07:20:23 GMT

Hi All,

I have a requirement where I need to load 10 CSV files in to one CSV file and from that one CSV file I need to load the table in Datawarehouse.

All the 10 CSV files have the same column structure and will not be changing.

Can you please let me know the job flow of how to achieve this.

Thank You.

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

TRF — Wed, 07 Nov 2018 20:47:39 GMT

3 main steps ti build the output file:
- tFileList to iterate over the 10 input files
- tFileInputDelimited to read the content of the current file using tFileList_1_CURRENT_FILEPATH to get the full pathname
- tFileOutputDelimited with append mode to generate the output file

Then you can get this file as the input to load data into your DWH. Details depend on the engine you are using for this DWH.

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

xx_emanis — Wed, 07 Nov 2018 20:51:35 GMT

Thanks for the quick reply.

Do I need to create 10 different metadata for the 10 different CSV I have?

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

TRF — Wed, 07 Nov 2018 20:54:09 GMT

Of course not as soon as all files have the same structure

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

xx_emanis — Thu, 08 Nov 2018 22:07:09 GMT

Thanks TRF I was able to acheive my requirement based on the flow you provided.

The other thing I was asked today was to create a LOAD_DATE column in the output CSV file and the load date should be based on the date which is a part of the name of the source CSV files.

For Instance name of one of the source csv files is All_VM_Cluster_Stats_2018-11-04.

When the data from this file is loaded to the target output csv file, LOAD_DATE column should be populated with '2018-11-04'

Please explain how to achieve this.

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

TRF — Thu, 08 Nov 2018 22:29:27 GMT

New question, thank's to open à new topic

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

xx_emanis — Thu, 08 Nov 2018 22:35:37 GMT

sorry dint get you..Do you want me to open a new topic for this question?

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

TRF — Thu, 08 Nov 2018 22:47:49 GMT

Yes, it's better to have just 1 question per topic

Re: Loading multiple CSV files to one CSV file and then to Datawarehouse table

Anonymous — Sat, 10 Nov 2018 14:34:11 GMT

- tFileList to iterate over the 10 input files
- tFileInputDelimited to read the content of the current file using https://fetlife.vip/ https://downloader.vip/imvu/ https://downloader.vip/canva/