Performance on a job

[Anonymous]

Hi,
For me, Talend is a very good solution but now, I try to do performance tests. And, my results are disastrous.
I join a screen of my job which discribe my business rules.
My project is to replace a loader done with Access.
With my current loader, the execution time of this job is 4minutes and 30 seconds.
With Talend, the execution time of this job is 1 hour and 28 minutes.
How can I improve the performance ?
Thx.

[Anonymous]

Hi suzchr,
There is a special reason using a tBufferOutput ? Can't you load directly in Access instead of loading in the buffer first ?

[Anonymous]

Yes, it's done but the best time get is 22 minutes. I try to use 2 jobs to improve this time.
After my test I can say that when you write on Access, the best commit value is 125 000 rows, but if I write on Access in the same job that I read in my data warehouse, with a commit of 125 000 rows the time is 6h 40minutes...

[Anonymous]

For help I add two screen shot which shows the two jobs.

[Anonymous]

Hi suzchr,
Try to test with only one tMap,it's accept many input and output ?
you may put a better Expression key to improve performence.
Regards

[Anonymous]

I don't have a key define on my schema. Do you think if I define a key I improve the performance ?

[amaumont]

Be careful, don't confuse between the key column into a schema and Expression key column into tMap.
Check the Key column from a schema will not improve performance into tMap, yet set an Expression key into one of your lookup will do.

[Anonymous]

Ok I define a key on my second TMap. I will send the results after.
Thx.

[Anonymous]

Have you tried to load a temporary table as output in the first job ?
Then you could map it with the output table in the 2nd job.

[Anonymous]

Hi,
Make sure that the metadata schema in yout tMap and in your tAccessOutput match exactly (including datatype) with the schema in your final database

[_AnonymousUser]

hi can some one tell me how to increase the speed of the rows(no.of.rows) passing in between source to destination...for eg...suppose if a csv file consists more than 1lakhs rows means while running the job..the rows passing in a speed of only 200 rows/s to 300rows/s...how to increase it to more than 1000rows/s...pls anyone reply me...