topic Re: How to copy data from S3 bucket to Redshift database in Talend Studio

How to copy data from S3 bucket to Redshift database

Anonymous — Wed, 29 May 2013 10:45:51 GMT

Hi
Which component I should use to copy the uploaded files on s3 bucket to the redshift database.If possible please share screenshot.

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Wed, 29 May 2013 11:13:59 GMT

Hi ashish.s,
So far, there is no Amazon S3 component in latest product(5.3.0). Please see the related forum https://community.talend.com/t5/Design-and-Development/Talend-to-Amazon-S3/td-p/108299 and vote for the jira issue Support to integrate with Amazon S3 (Simple Storage Service)
Best regards
Sabrina

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Wed, 29 May 2013 14:02:27 GMT

Hi sabrina
i got s3 component through this http://www.labdecisionnel.com/index.php?option=com_content&view=section&layout=blog&id=12&Itemid=144&limitstart=5
Ysance_ComposantsS3.zip file I installed it in my studio but i am not finding any component by which we can write copy command like( copy events
from 's3://$MY-BUCKET/PATH/TO/FILES/FOR/UPLOAD'
credentials 'aws_access_key_id=$ACCESS-KEY;aws_secret_access_key=$SECRET-ACCESS-KEY'
delimiter '\t' as per the AWS doc to load data in redshift database
Thanks
Ashish.s

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Thu, 30 May 2013 06:43:44 GMT

Hi,

i got s3 component through this http://www.labdecisionnel.com/index.php ? mitstart=5
Ysance_ComposantsS3.zip file I installed it in my studio but i am not finding any component by which we can write copy command like( copy events
from 's3://$MY-BUCKET/PATH/TO/FILES/FOR/UPLOAD'
credentials 'aws_access_key_id=$ACCESS-KEY;aws_secret_access_key=$SECRET-ACCESS-KEY'
delimiter '\t'wink as per the AWS doc to load data in redshift database

It neither is a talend component nor is talend custom component. So have no idea about that. I have made a search on talend exchange portal and found some custom components about S3 http://www.talendforge.org/exchange/index.php?page=2&product=tos&nav=1,1,1, perhaps it will give a help to you. However, Components available in Talend Exchange are developed by the Community thus are not available natively in the Studio, you need to install them.
Best regards
Sabrina

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Thu, 30 May 2013 07:40:38 GMT

Hi
Solved the problem through using tpostgresqlrow component and added copy command in query.

Thanks
Ashish.s

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Tue, 04 Jun 2013 16:57:19 GMT

Hi,
And by the way, you have the tRedshiftRow which allows to do the same thing.
Cheers,

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Fri, 23 Aug 2013 00:57:51 GMT

Hi
Solved the problem through using tpostgresqlrow component and added copy command in query.

Thanks
Ashish.s

hey how did you write the copy query.did u write through the query or the copy command in cmd

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Tue, 27 Aug 2013 21:33:45 GMT

Thanks it worked

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Tue, 27 Aug 2013 21:42:12 GMT

!SESSION 2013-08-27 09:26:57.443 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.7.0_25
java.vendor=Oracle Corporation
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Command-line arguments: -os win32 -ws win32 -arch x86
This is a continuation of log file C:\Talend\TOS_DI-Win32-r104014-V5.3.1\workspace\.metadata\.bak_0.log
Created Time: 2013-08-27 09:27:00.579
!ENTRY org.eclipse.osgi 2 0 2013-08-27 09:27:00.579
!MESSAGE The following is a complete list of bundles which are not resolved, see the prior log entry for the root cause if it exists:
org.osgi.framework.BundleException: Exception in org.eclipse.core.resources.ResourcesPlugin.start() of bundle org.eclipse.core.resources.
Root exception:
org.eclipse.core.internal.dtree.ObjectNotFoundException: Tree element '/SAS_STOCKCHECK/process/SFTP_ETL_MIGRATION_0.1.screenshot' not found. at org.ecli

To see the whole post, download it here
OriginalPost.pdf

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Tue, 27 Aug 2013 21:47:54 GMT

I am running into trouble when I am running TOS on server. On local host it is running fine..Can anyone look into it and let me know the solution soon or should I re install it
!ENTRY org.eclipse.osgi 4 0 2013-08-27 09:29:47.387
!MESSAGE Application error
!STACK 1
java.lang.RuntimeException: The service org.talend.core.model.migration.IMigrationToolService has not been registered.
at org.talend.core.GlobalServiceRegister.getService(GlobalServiceRegister.java:148)
at org.talend.rcp.intro.Application.start(Application.java:84)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:619)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
at org.eclipse.equinox.launcher.Main.run(Main.java:1407)

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Tue, 10 Dec 2013 20:27:20 GMT

My Talend DI 5.3 keeps on crashing after I create some jobs. I had to reinstall fresh copy again. This is the error in the log file. I am running this on Windows 8.1
!MESSAGE An error occurred while automatically activating bundle org.eclipse.emf.ecore (380)
PLEASE HELP!!!!

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Wed, 11 Dec 2013 03:29:34 GMT

Hi Max_Hamlet,
You'd better open a new forum for your topic.
For your scenario, did you use any custom component in your specific jobs? Is it normal before? Is there any more error log which is saved in <Talend Studio installation path>/workspace/.metadata/.log.
Best regards
Sabrina

Re: How to copy data from S3 bucket to Redshift database

_AnonymousUser — Mon, 08 Aug 2016 12:57:05 GMT

Hi
Solved the problem through using tpostgresqlrow component and added copy command in query.

Thanks
Ashish.s

Hi Ashish,
Can you please provide the syntax which you have used to copy the data. Writing simple copy command gives an error of Must be a super user to execute copy operation.
Thanks
Ravi

Re: How to copy data from S3 bucket to Redshift database

Anonymous — Wed, 10 Aug 2016 12:08:10 GMT

if you're using Talend 6 and above , Talend supports Bulk Load using S3 natively.
I have put an example and a comparison on the performance by using S3 and without it ( JDBC )
https://talendexpert.com/using-talend-load-data-redshift/
The component you should us for Bulk Load is : tRedshiftOutputBulkExec