topic Re: million records in Talend Studio

million records

shrikavi2008 — Tue, 19 Jan 2016 16:07:55 GMT

Hi,
I have few questions
Q1 . Can we able to process 60-80 million records in Talend ?
Q2. What is the best way on performance wise , if I have 4-6 million records on lookup ?
Q3. which is best option to store lookup data, if I have 4-6 million look up records ? either file or DB ?
Regards.

Re: million records

Anonymous — Wed, 20 Jan 2016 07:04:57 GMT

Hi,
There is no standard answer for the maximum volume of data handled by Talend. It depends on project scale, job design, data source and so on.
Do you want to use SQL query or tMap?making a join? to handle Lookups?
Please provide the details. ELT components(all tables are in same DB), bulk execute are better way to load large data in a faster way.
Best regards
Sabrina

Re: million records

Anonymous — Wed, 27 Apr 2016 16:51:07 GMT

Probably a late reply, but it might useful for others.
Q1. Assuming that you are reading from RDBMS Table: Yes, but it should be in stream mode to avoid Java heap error. I have loaded 33 million records extracted from 1 billion records table in MySQL through Talend.
Q2. For better performance, we need to consider the total number of input data and output records count. For the above scenario, I need only 33 millions of records out of 1 billion records. I used a inner join query in extract query with stream mode, instead of lookup. Look up table's data needs to reside in the Server RAM. So, size of lookup table/file depends on the Server RAM, on which Talend installed.
Q3. I advice you to store the lookup data in the DB and join them in the extract query, as the lookup data is huge.
Thanks,
Srini,
AgilitX