Long Job execution time

[vidya821]

Hi ,

I am processing a delimited file with 450,000 records to be inserted in a table.

Table has necessary indexes.

its simple job with reading the data , mapping and inserting via OracleSCD component

I have started the exeution of the job and its been more than 6 hours and it has processed 320,000 records.

Is this how normally talend job is executed or should it be much faster. ?

Attached is the job design .

Also attached is the oracleSCD component properties.

Could you please provide me some pointers to look into the issue (if its an issue)

Thanks.

[vapukov · Accepted solution]

it is not dedicated for Talend problem, it would be same for any database - plain insert work fast, insert with update work 1000 times slower

so and with Talend - all what compare data before insert would be work much slower than just insert, plus of course - jdbc, network and etc add some delay

in Your case - 15 records per second, more or less similar with my results for MySQL and MS SQL

[vidya821]

Thanks,
It seems there is no solution for the problem (since the DB and connections are all good enough), but this is a concern since i have to process 5 to 6 files of same size in a single job.

[Anonymous]

Hi,

Are you facing this long Job execution time issue on talend subscription solution?

Best regards

Sabrina

[vidya821]

Hi,

I am sorry but what is talend subscription solution ?

I am using Talend Open Studio for Data Integration V6.4.0

Thanks

[vapukov]

Sabrina mean - Talend commercial version

generally, same Jobs on Talend server could work faster, it could be affected by network topology and server configuration, but not dramatically

but this difference not resolve described problem

[vidya821]

Thanks,

For OracleSCD component to process faster , is it advised to create index on type 2 fields in DB table ?
Note: the no of fields in type 2 are around 10

[naveenhks]

SCD on talend is very challenging. It is not worth spending time in building an talend component with SCD. Same logic with stored proc runs 10 time faster than talend job. Talend job was running on same box where the stored proc was executed on a test box to eliminate network overhead. 

[vapukov]

---

@nhalicka wrote:

SCD on talend is very challenging. It is not worth spending time in building an talend component with SCD. Same logic with stored proc runs 10 time faster than talend job. Talend job was running on same box where the stored proc was executed on a test box to eliminate network overhead. 

---

you are right about SCD component (ETL) which is slow because execute INSERTS and UPDATES for each row, but SDCELT component (present for each database where supported SCD plus universal JDBC) work with the same speed s stored procedure because generate proper queries right on the target database.

Unfortunately, SCDELT required both tables (staging and target) in the same schema.