Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements
See why IDC MarketScape names Qlik a 2025 Leader! Read more
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable
‎2014-06-11 09:29 AM

Talend & Redshift AWS

Hi All,
i'm currently using talend to transfer my data from Oracle Database to my Redshift Cloud database.
what I noticed is that it is really slow, with an average of 240 row / second on only three fields.
Is there a trick to improve the transfer?
Labels (3)
Labels
651 Views
0 Likes
9 Replies
Anonymous
Not applicable
‎2014-06-12 12:06 AM
Author

Hi n.kasdali,
Performance issue is usually caused by the DB connection or the job design, can you upload some screenshots of job design into forum?
Best regards
Sabrina
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 05:35 AM
Author

hi, i've got 6gb/s UP bandwidth in my network, and 3.6gb/s using the aws network.
For my Job is quite simple:
tOracleInput---->tMap-------->tRedshiftOoutput.
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 05:47 AM
Author

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenterutOfMemory.
Best regards
Sabrina
0683p000009MEOc.png
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 06:25 AM
Author

At first, I started by doing a CSV extraction.
However, I have a field "USER_AGENT" which contains a wealth of character, which makes it difficult to use a delimiter.
This is why I use a tMap.
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 06:27 AM
Author

Hi,
Is "Extend Insert" option in Advanced Setting of tRedshiftOutput OK with your scenario?
Best regards
Sabrina
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 06:50 AM
Author

i've got the same option : commit every 10000 and insert ligne 100.
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 07:07 AM
Author

Hi,
Have you selected "Extend Insert" check box to carry out a bulk insert of a defined set of lines? TalendHelpCenter:tRedshiftOutput.
In addition, tMap is cache component consuming two much memory. For a large set of data, try to store the data on disk instead of memory on tMap. Also, allocate more memory to execute the job.
Please have a look at KB article TalendHelpCenter0683p000009MA5A.pngutOfMemory.

Does these solutions make your job performance improved?
Best regards
Sabrina
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 07:11 AM
Author

i'll try 0683p000009MA9p.png
651 Views
0 Likes
Anonymous
Not applicable
‎2014-06-12 07:14 AM
Author

Hi,
Feel free to let me know if is it OK with you.
Best regards
Sabrina
651 Views
0 Likes