Job Performance tuning help required incase of genarting extract file from sybase table

[Anonymous]

Hi,

I am simply generating extract file from sybase table using tSybaseInput and tfileoutputdelimited using tmap and i am not doing any transformation. Table have data around 217 million. can someone help  me to provide performance technique which i should use to get better performance. I am using Talend 6.5.1.

Regards,

Rahul

[ankit7359]

Hi @rahuljan ,

You can use parallelization and increase the number the of child process(threads) , also what is the exact time the job is running.???

Pls do get to Community, if necessary.

Thanks,

Ankit.

[Anonymous]

Hi Rahul,

     One way of increasing the throughput of writing data is to use tparallelize and fetch the data in parallel for diferent data sets from source database. For example, you can run two or three subjobs from tparallelize component which fetches data for different date range and writing to target file. Please remember to add the append mode on so that multiple proccess can write at same time.

     You will have to increase the memory parameters also for the job to make sure that data is passing without any hiccups.

    I would even try to run multiple jobs also to make things faster by using all available firepower in Jobserver 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

@nthampi  Is there any other solution since using parallel component data will be not in same sequence and that i important in my job?

[dipanjan93]

Check the below url -

Custom the flush buffer size

This is usually there in the Advanced Settings Tab of tFileOutputDelimited

[Anonymous]

Hi,

    I agree with @dipanjan93  about buffer size flusing as it is also going to change your data throughput.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

Hi @rahuljan ,

How about the query performance if we use limit clause. If it is good, then we can export data without using tParallel component.

Ex Query: 

select * from tablename limit 1000000;

[Anonymous]

@hpamidipola 

Using Limits might not be a good idea in this case as we have to export the data from source table even if the records are more than the limit level. If we specify a hardcoded limit, the data after the limit will be ignored and will eventually result in data loss or mismatch.

@rahuljan 

  If the details provided by the members have helped you, could you please mark the topic as resolved? It will help other community members also during their reference.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂