Good day,
I am writing this to understand more about Talend products mentioned above. The main purpose I am asking this is because the webinar I?ve attended were not using open source, there were on TIS / TDQ. So, using TOS is it able to achieve same performance of transferring rate for (heterogeneous & homogenous environment)?
We experienced slow transferring rate using JasperETL Pro for large datasets ie: >2 millions of records. See below:
Heterogeneous environment benchmark testing (oracle to mysql)
Total Number of Rows: 1095934
Number of Rows per second: 617.03
AVG time in mints: 29.6
Total Number of Rows: 2025100
Number of Rows per second: 159.12
AVG time in mints: 212.11(3.53hr?s)
Homogenous environment benchmark testing (mysql to mysql)
Total Number of Rows: 1095934
Number of Rows per second: 4345.24
AVG time in mints: 4.20
Total Number of Rows: 2020000
Number of Rows per second: 595.88
AVG time in mints: 56.49
Does Talend having benchmark test result to share with us? I afraid there are 1 or more settings we could miss out that result poor performance?
I seek clarification on this to clear our doubts. Furthermore, would it be possible for us to get evaluation product for TIS & TDQ?
It would be highly appreciated for your input to shine our direction.
Hi Cedric Carbone, Thanks for your reply and please find my screenshot for the job prepared on benchmark testing. Yes. For source table we selected (enable stream). For target table we selected (extend insert). We are here using Jasper ETL professional, which the same as Talend TOS + AMC. Regards, Yoke Yew
Hi, Your performance is very bad. What about the "commit every x rows" field? Can you try with a bulk load? Are the 2 DB on the same server? If not, what is the kind of network do you use?
Hi Cedric Carbone,
The are 2 environments mentioned. Homogenous (mysql -> mysql) happen in the same machine.
Heterogenous (oracle -> mysql) is on 2 different servers. We are using 100mbps LAN. Please comment on this further.
ToDo:
1)We will provide you another result for "commit every x rows" field.
2)Regarding bulk load, is it "t000BulkExec" where 000=mysql, oracle...I understand that is loading data from a file but our design is loading data from 1 RDBMS to another RDBMS.
Regards,
Yoke Yew
2)Regarding bulk load, is it "t000BulkExec" where 000=mysql, oracle...I understand that is loading data from a file but our design is loading data from 1 RDBMS to another RDBMS.
Hi Cedric Carbone,
For item as discussed above, please find details result in the diagram below.
1) We are testing on 2 different server2 (Oracle -> Mysql) using 100mbps LAN. The new setting is commit every 1 row.
2) We are not using Load Data Infile. Therefore we did not test on t000OutputBulkExec. Most of situation we will transfer datasets involve 2 different RDBMS server (source, target)
Please advise further.
1) We are testing on 2 different server2 (Oracle -> Mysql) using 100mbps LAN. The new setting is commit every 1 row.
Can you set up the commit every X rows field to 50000 or 100000.
2) We are not using Load Data Infile. Therefore we did not test on t000OutputBulkExec. Most of situation we will transfer datasets involve 2 different RDBMS server (source, target)
I don't understand your answer. Our tMySQLOutputBulkExec runs good even if you are on 2 different RDBMS server. Have you try this method, I think it's the best method for performance.
Hi Cedric Carbone, For item as discussed above, 1) We have tried setting commit every x rows field with "50,000" and it is slower than the default value "10,000" (see the 1st result prepared above) 2) Using tMySQLOutputBulkExec, it is only allow to output to csv file. We are unable to output to MySQL DB. Could you provide a sample for us to refer? Please advise further. Regards, Yoke Yew
2) Using tMySQLOutputBulkExec, it is only allow to output to csv file. We are unable to output to MySQL DB. Could you provide a sample for us to refer?
tMySQLOutputBulkExec apply 'bulk insert' way when loading records into db, you could read the more demonstration on
user documenation. Here is a simple demo. see my screenshot.
Best regards
