topic Connecting to Amazon S3 Storage in Talend Studio

Connecting to Amazon S3 Storage

Anonymous — Sat, 16 Nov 2024 12:13:54 GMT

Hi,
I am very newbie to Talend and trying to connect to Amazon S3 (simple storage service). Please let me know the steps to do.
Steps tried:
1. I've the following version Talend installed
"Talend Open Studio for Big Data"
Version: 5.1.1
Build id: r84309-20120530-0555
2. I've installed http://www.talendforge.org/exchange/tos/extension_view.php?eid=394&rid=645#rev645 component and kept it part of local components section.
3) While trying to connect to S3 using tS3connection component,
Access Key - <username of my AWS account - My user format is like test.test.com)
Secret Key - <password of my AWS account>
While execute by Run (Job Demo), getting the following error.
Module amazonS3lib.jar required
Moudle ysanceS3lib.jar required
But these jars are already available in the same directory itself.

$ ls -l
total 91
-rwx------+ 1 Administrators mkgroup 42760 Mar 23 2010 amazonS3lib.jar
-rwx------+ 1 Administrators mkgroup 751 Mar 23 2010 tS3Connection_begin.javajet
-rwx------+ 1 Administrators mkgroup 665 Mar 22 2010 tS3Connection_begin_2.javajet
-rwx------+ 1 Administrators mkgroup 3771 Jun 29 2009 tS3Connection_icon32.png
-rwx------+ 1 Administrators mkgroup 3657 Jun 29 2009 tS3Connection_icon32_white.png
-rwx------+ 1 Administrators mkgroup 1427 Jun 20 2011 tS3Connection_java.xml
-rwx------+ 1 Administrators mkgroup 110 Mar 23 2010 tS3Connection_messages.properties
-rwx------+ 1 Administrators mkgroup 108 Mar 23 2010 tS3Connection_messages_en.properties
-rwx------+ 1 Administrators mkgroup 107 Mar 23 2010 tS3Connection_messages_fr.properties
-rwx------+ 1 Administrators mkgroup 24045 Mar 24 2010 ysanceS3lib.jar
Let me know the steps to resolve this. Thanks.
--
Muthu

Re: Connecting to Amazon S3 Storage

Anonymous — Fri, 27 Jul 2012 02:57:35 GMT

Hi Muthu
Because this custom component is not supported by Talend officially and the compitable version is not 5.1.1, I can't make sure it would work fine.
To solve the error you got, try to copy these two jars into <tos installation dir>\lib\java.
Hope this would help you.
Regards,
Pedro

Re: Connecting to Amazon S3 Storage

Anonymous — Fri, 27 Jul 2012 07:41:50 GMT

Hi Pedro,
Thanks for your prompt support and work around.
I've copied the jar files into <tos_install_dir>/lib/java and able to surpass the error. Now I am getting the following issue as,
Starting job demo at 12:06 27/07/2012.

connecting to socket on port 3460
Exception in thread "main" java.lang.Error: Unresolved compilation problems:
test cannot be resolved to a variable <<< where test is part of my Amazon AWS Account Name
Syntax error on token "@", . expected <<<< **** is my amazon AWS secure key
****** (securekey) cannot be resolved to a variable
at verizontest.demo_0_1.demo.tS3Connection_1Process(demo.java:253)
at verizontest.demo_0_1.demo.runJobInTOS(demo.java:492)
at verizontest.demo_0_1.demo.main(demo.java:360)
connected
Job demo ended at 12:06 27/07/2012.
Please let me know the steps or guideline to handle this. And it will be good to know the authorized or approved way of connecting to Amazon S3 (storage) to read/write/delete/update operations.
Let me know if more details are required. Your help will be really appreciated. Thanks.
--
Muthu

Re: Connecting to Amazon S3 Storage

Anonymous — Fri, 27 Jul 2012 07:53:07 GMT

Hi Muthu
I think this custom component has bugs.
Besides, for now, there isn't any Talend official component to support S3.
For any new feature, please report it on BugTracker.
Regards,
Pedro

Re: Connecting to Amazon S3 Storage

Anonymous — Mon, 30 Jul 2012 05:39:55 GMT

Hi Pedro,
Thanks for your great response.
I've filed a JIRA now (Ref: TDI-22143).
--
Muthu

Re: Connecting to Amazon S3 Storage

Anonymous — Mon, 30 Jul 2012 05:46:32 GMT

Hi Muthu
Thanks for your feedback!
Regards,
Pedro

Re: Connecting to Amazon S3 Storage

Anonymous — Fri, 05 Apr 2013 10:50:42 GMT

Hi,
I have recently started working on Talend, I want to get data through Amazon S3 bucket.
Is there any way to get amazon S3 bucket data through Talend?

Re: Connecting to Amazon S3 Storage

Anonymous — Fri, 05 Apr 2013 14:10:53 GMT

Hi kmuthu_gct,
As you posted above-------
************************************************************************
Hi,
I am very newbie to Talend and trying to connect to Amazon S3 (simple storage service). Please let me know the steps to do.
Steps tried:
1. I've the following version Talend installed
"Talend Open Studio for Big Data"
Version: 5.1.1
Build id: r84309-20120530-0555
2. I've installed http://www.talendforge.org/exchange/tos/extension_view.php?eid=394&rid=645#rev645 component and kept it part of local components section.
3) While trying to connect to S3 using tS3connection component,
Access Key - <username of my AWS account - My user format is like test.test.com)
Secret Key - <password of my AWS account>
While execute by Run (Job Demo), getting the following error.
Module amazonS3lib.jar required
Moudle ysanceS3lib.jar required
But these jars are already available in the same directory itself.

$ ls -l
total 91
-rwx------+ 1 Administrators mkgroup 42760 Mar 23 2010 amazonS3lib.jar
-rwx------+ 1 Administrators mkgroup 751 Mar 23 2010 tS3Connection_begin.javajet
-rwx------+ 1 Administrators mkgroup 665 Mar 22 2010 tS3Connection_begin_2.javajet
-rwx------+ 1 Administrators mkgroup 3771 Jun 29 2009 tS3Connection_icon32.png
-rwx------+ 1 Administrators mkgroup 3657 Jun 29 2009 tS3Connection_icon32_white.png
-rwx------+ 1 Administrators mkgroup 1427 Jun 20 2011 tS3Connection_java.xml
-rwx------+ 1 Administrators mkgroup 110 Mar 23 2010 tS3Connection_messages.properties
-rwx------+ 1 Administrators mkgroup 108 Mar 23 2010 tS3Connection_messages_en.properties
-rwx------+ 1 Administrators mkgroup 107 Mar 23 2010 tS3Connection_messages_fr.properties
-rwx------+ 1 Administrators mkgroup 24045 Mar 24 2010 ysanceS3lib.jar
Let me know the steps to resolve this. Thanks.
--
Muthu
************************************************************************

I didnt find any tS3connection component in Talend exchanges,I searched there and found only one component named tS3ObjectUpload.
Is it the same component you used?
I want to retrieve information from Amazon S3 bucket through Talend
and when I am using this component it is showing some jar file is missing, I didn't find any jar file within this component folder.
1.ysanceS3lib.jar
2.zookeeper-3.3.3cdh3u0.jar
these two jar files are missing in that component.
where will i find these jar files? I tried to search these files but didnt get it.
Please help.

Thanks,
Anand

Re: Connecting to Amazon S3 Storage

Anonymous — Fri, 05 Apr 2013 16:26:52 GMT

Hi
When I connect to http://www.talendforge.org/exchange/tos/extension_view.php?eid=394&rid=645#rev645 it says the connectors are compatible with Talend Open Studio 5.0.2, and you said you downloaded v5.1.1, maybe there is an issue here.
In the newer version of Talend (5.2.2) documentation there is a section that explains how to install external jars, this may help: https://help.talend.com/search/all?query=Installing+external+modules&content-lang=en
But I'm not sure this is available in the v5.1.1 though.

Elisa

Re: Connecting to Amazon S3 Storage

Anonymous — Wed, 08 Jan 2014 03:26:26 GMT

try ts3bucketexists

Re: Connecting to Amazon S3 Storage

_AnonymousUser — Fri, 15 Apr 2016 10:40:00 GMT

Hi,
Just want to know,
There are connectors available for Amazon S3 in Talend Open Studio for Data Integration (version 6.0) also.
So can i create the connections and do the needeful in Data Integration also or is it that I require Talend Open Studio for Big Data only
I am confirming this because all the articles on web are talking about Talend Open Studio for Big Data only