topic Re: How to handle a large SAP table efficiently in Talend Studio

How to handle a large SAP table efficiently

AndrewGuy — Wed, 23 Jan 2019 12:35:29 GMT

Task : Extract all the records from a large sap table and store in a SQL table - 39,000,000 records.

Progress: I have created a job with a tSAPTableInput component , one tmap and a tMSSQLOutput component ( so far so good).

Problem: The performance starts well but then settles into 25 records per second - this will take days to complete.

Question : Is there a way to improve performance or perhaps break it into chucks effectively ?

Re: How to handle a large SAP table efficiently

Anonymous — Wed, 23 Jan 2019 13:01:13 GMT

Hi,

Did you try tparallelize as the starting point and from there multiple data fetch flows going to output location?

Please increase the memory of the job so that Talend job can process without any hindrance.

If its still taking more time, write extracted data to file in parallel mode and then use tDBBulk components instead of tDBOutput component to process data faster.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: How to handle a large SAP table efficiently

AndrewGuy — Tue, 12 Feb 2019 14:26:50 GMT

Dear Nikhil ,

The tDBBulk component is not available at my version. The tParallelize component did certainly speed things up but it still crashes with memory issues.

Re: How to handle a large SAP table efficiently

Anonymous — Tue, 12 Feb 2019 15:02:09 GMT

Hi,

Could you please cross check the Xms and Xmx parameters of your Talend job and fine tune them according to your system resource availability?

The fine tuning of these parameters will help you to avoid crash due to memory issues. Detailed information can be fetched from below link. You can change these parameters under Advanced section of Run tab.

https://help.talend.com/reader/_zCTtDbUbgf9AGOJY8pz1Q/EoxO17sMiUedPquMxGLGdg

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂