topic Extract a billion records using Talend Data Integration in Talend Studio

Extract a billion records using Talend Data Integration

Anonymous — Thu, 05 Dec 2013 02:32:47 GMT

Hi,
May i know is this TOS for data integration might have a capability to extract a billions records?
I need to extract out a billions of records from Oracle/db2 database.
Anyone have any experience on this?
Regards,
Ommie

Re: Extract a billion records using Talend Data Integration

Anonymous — Thu, 05 Dec 2013 06:13:42 GMT

Hi,
Here is a similar topic https://community.talend.com/t5/Design-and-Development/Data-Size-handled-by-Talend-open-studio-for-Data-integration/td-p/64778.
Best regards
Sabrina

Re: Extract a billion records using Talend Data Integration

Anonymous — Sat, 18 Jan 2014 11:55:23 GMT

Talend does not have a limitation in processing number of row. You can process as many rows as you want/have. It is always a matter of efficiency and resources. Billions of row - why not.
I would always try to separate a chunk of rows and avoid trying to process all rows at once.
Processing billions of rows without separating it in chunks will lead into a nightmare on the productive system - regardless which tool do you use to do that.