topic Re: Talend & Redshift AWS in Talend Studio

Talend & Redshift AWS

Anonymous — Sat, 16 Nov 2024 11:36:56 GMT

Hi All,
i'm currently using talend to transfer my data from Oracle Database to my Redshift Cloud database.
what I noticed is that it is really slow, with an average of 240 row / second on only three fields.
Is there a trick to improve the transfer?

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 04:06:21 GMT

Hi n.kasdali,
Performance issue is usually caused by the DB connection or the job design, can you upload some screenshots of job design into forum?
Best regards
Sabrina

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 09:35:20 GMT

hi, i've got 6gb/s UP bandwidth in my network, and 3.6gb/s using the aws network.
For my Job is quite simple:
tOracleInput---->tMap-------->tRedshiftOoutput.

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 09:47:03 GMT

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterutOfMemory.
Best regards
Sabrina

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 10:25:07 GMT

At first, I started by doing a CSV extraction.
However, I have a field "USER_AGENT" which contains a wealth of character, which makes it difficult to use a delimiter.
This is why I use a tMap.

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 10:27:33 GMT

Hi,
Is "Extend Insert" option in Advanced Setting of tRedshiftOutput OK with your scenario?
Best regards
Sabrina

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 10:50:55 GMT

i've got the same option : commit every 10000 and insert ligne 100.

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 11:07:20 GMT

Hi,

Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterutOfMemory.

Does these solutions make your job performance improved?
Best regards
Sabrina

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 11:11:58 GMT

i'll try

Re: Talend & Redshift AWS

Anonymous — Thu, 12 Jun 2014 11:14:25 GMT

Hi,
Feel free to let me know if is it OK with you.
Best regards
Sabrina