topic Re: Inserting data to redshift using tredshiftoutputbulkexec -->slow loading in Talend Studio

Inserting data to redshift using tredshiftoutputbulkexec -->slow loading

Anonymous — Sat, 16 Nov 2024 03:39:50 GMT

Initially I tried tmongodbinput ==> tredshiftoutput the data loaded succesfully, but it took me about 460 rows/second it takes too long time(12 mins). To improve the performance tried tredshiftoutputbulkexec component.

Now trying to load my mongodb data into the redshift (3 lakhs data). Going from (tmongodbinput) directly to (tredshiftoutputbulkexec) got me about same time approx 450 rows/second(12-13 mins). tredshiftoutputbulkexec also takes same time to load data. how to improve the performance?

Reading the data is faster in tredshiftoutputbulkexec but commits takes same as tredshiftoutput.

I attached the screen shot below.

-----------------------------------------------

Can anyone please help me with the solution.

Please do let me know if you need any more details.

Re: Inserting data to redshift using tredshiftoutputbulkexec -->slow loading

fdenis — Tue, 07 Jan 2020 09:26:10 GMT

mongo and redshift data… are they on hosted machines… how is the network between them. if there is no speed variations.. the trouble may be on server speed….

Re: Inserting data to redshift using tredshiftoutputbulkexec -->slow loading

MattE — Tue, 07 Jan 2020 09:39:00 GMT

Is it necessary to drop the table and recreate each time? Admittedly that shouldn't make the query run at 12 minutes.

The only other thing i have in my redshift setup is that in advanced i tick the Compressed by - gzip checkbox. You could log into AWS cloudwatch and view the queries that are being executed as the job runs, this might give you an idea if the problem is on the redshift side or not.