Performance on a job

[Anonymous]

Hi,
For me, Talend is a very good solution but now, I try to do performance tests. And, my results are disastrous.
I join a screen of my job which discribe my business rules.
My project is to replace a loader done with Access.
With my current loader, the execution time of this job is 4minutes and 30 seconds.
With Talend, the execution time of this job is 1 hour and 28 minutes.
How can I improve the performance ?
Thx.

[Anonymous]

Hi,
Some questions :
- How many rows are treated by your job ?
- Do you launch routines in your tMap ?
- Your databases are local or on network ?
And an experience :
Before launching the job, check the "Statistics" box to see where the dataflow is slow

[Anonymous]

There are 421 000 rows on my job.
Yes I launch some routines in my two tmap but not complex.
My database Oracle is on network and access is local.
It's exactly the same configuration that my other loader.
More precisions : on my job the tAccessInput and the tAccessOutput is the same database Access.

[Anonymous]

Before launching the job, check the "Statistics" box to see where the dataflow is slow
With this, you'll be able to see where things are slow (is it on oracle or access ??)
Once you have check with statistics, try to separate access input and output in two files

[Anonymous]

Show the statistics is not very interisting because the stat are identical in all the job about 120 rows/seconds

[Anonymous]

it is interesting, it shows that dataflow is slowing at the entry point : your oracle database on network
Could you show a screenshot of your tOracleInput component's configuration ?

[Anonymous]

Precision :
Oracle is connect by ODBC. I try by JDBC and it's about the same perfs.
Access is local : my file is on my computer.
Oracle is distant is on my network.
I add a screen of my param in my first post. See top.

[Anonymous]

Talend Open Studio generate Java or Perl code.
None of these language manage Access database natively.
Java Access DB Components communicate through an ODBC Bridge.
It will never be as fast as a native connexion or as fast as a real JDBC connexion.
Generally speaking, Input is not the problem. Writing is always longer...
That's why when you change tOracleInput, to ODBC it doesn't change the results.
You can try to tweak Avanced Settings / Autocommit value in your tAccessOutput : increase this value.
HTH,

[Anonymous]

For me, it's much faster to read/write an SQL database than a flat file (I can reach 40000 rows/sec against 3000 for a flat file)
All my jobs use Java

[Anonymous]

I only said that reading is faster than writing.
About writing to file, on my laptop with a quite slow hard disk, I can easily go up to 75000 rows per second.
My Input file has 1 000 000 rows and 11 columns with different data types (Integer, String, and Date).
I write to a simple tFileOuptutDelimited without any CSV options...
My only "special" configuration is to temporary disable my Antivirus.
Can you give me more details about your own tests ?