How to handle a large SAP table efficiently

[AndrewGuy]

Task : Extract all the records from a large sap table and store in a SQL table - 39,000,000 records. 

Progress: I have created a job with a tSAPTableInput component , one tmap and a tMSSQLOutput component ( so far so good). 

Problem: The performance starts well but then settles into 25 records per second - this will take days to complete. 

Question : Is there a way to improve performance or perhaps break it into chucks effectively ? 

[Anonymous]

Hi,

    Did you try tparallelize as the starting point and from there multiple data fetch flows going to output location?

    Please increase the memory of the job so that Talend job can process without any hindrance.

     If its still taking more time, write extracted data to file in parallel mode and then use tDBBulk components instead of tDBOutput component to process data faster.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[AndrewGuy]

Dear Nikhil , 

The tDBBulk component is not available at my version. The tParallelize component did certainly speed things up but it still crashes with memory issues. 

[Anonymous]

Hi,

   Could you please cross check the Xms and Xmx parameters of your Talend job and fine tune them according to your system resource availability?

    The fine tuning of these parameters will help you to avoid crash due to memory issues. Detailed information can be fetched from below link. You can change these parameters under Advanced section of Run tab.

https://help.talend.com/reader/_zCTtDbUbgf9AGOJY8pz1Q/EoxO17sMiUedPquMxGLGdg

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂