topic Re: Aggregation on large Table Data in Talend Studio

Aggregation on large Table Data

Anonymous — Sat, 16 Nov 2024 10:39:52 GMT

I have a table containing 33 million records in MySQL database. I need to aggregate the rows. Could you help to know the best/efficient method to solve this with performance?
1. Extract all the 33 million records into Talend, sort and aggregate in the Talend tool
2. TMySQLInput with following extract query
   SELECT A,B,C, SUM(D),SUM(E)
   from table1
   group by A,B,C
3. Any other better way other than listed above
Thanks,
Srini

Re: Aggregation on large Table Data

Anonymous — Wed, 04 May 2016 09:09:00 GMT

Hi,
Here is a component TalendHelpCenter:tAggregateRow which receives a flow and aggregates it based on one or more columns. For each output line, are provided the aggregation key and the relevant result of set operations.
Please take a look at the related scenario in this component reference.
Best regards
Sabrina

Re: Aggregation on large Table Data

David_Beaty — Tue, 04 Dec 2018 16:54:31 GMT

Hi,

A combination of tMySQLInput (with order by criteria) and tAggregateSortedRow components together. Make sure you correctly set the "Input Rows Count" which means you'll probably need to read the data into an intermediate file.

tCreateTemporaryFile

tMySQLINput -> tFileOutputDelimited

tFileInputDelimted -> tAggregateSortedRow -> (whatever you want to do with the aggregated data)

Thanks

David