Data insertion performance using Talend DI Jobs

Anonymous — Fri, 17 Jun 2016 08:03:03 GMT

Talend version - 6.1.1.
SVN Version - svn, version 1.6.11 (r934486)
CentOS release 6.7 (Final)
DB - MySQL 5.7.13 InnoDB Engine
Infrastructure hosted in Azure Cloud
Environment - Pre-PROD

We are facing performance issues while inserting the data into MySQL DB using the Talend jobs deployed in TAC.
Problem Statement :
# Job design - with only insert in tmySQLoutput component, data flow rate is >2000rows/sec.
# Job design - with only update in tmySQLoutput component, data flow rate is >2000rows/sec
# Job design - with insert or update in tmySQLoutput component, data flow rate is very poor. approx 3-4rows/sec.

We are having the same architecture in development environment. in DEV env, even jobs with design as insert or update in tmySQLoutput component, data flow is far better >1000 rows/sec.

Talend DI Server & MySQL DB versions are same in both the environments. But facing the data flow issue in Pre-PROD env.

Can anyone provide help on this issue.

Thanks
Ashok Karnam

Re: Data insertion performance using Talend DI Jobs

Anonymous — Mon, 20 Jun 2016 04:18:20 GMT

Hi,
What does your job design look like? Have you already checked out "Extend Insert " option on tMySqlOutput component to carry out a bulk insert of a defined set of lines?
Have you already reported a ticket on Talend Support Portal for your subscription version? In this way, we can give you a remote assistance through support cycle with priority.
Best regards
Sabrina

topic Re: Data insertion performance using Talend DI Jobs in Talend Studio

Data insertion performance using Talend DI Jobs

Re: Data insertion performance using Talend DI Jobs