topic Issue with tS3List component in Talend Studio

Issue with tS3List component

_AnonymousUser — Tue, 14 Oct 2014 17:14:59 GMT

I'm currently using Talend Open Studio Data Integration 5.5.1. I've created a job that first creates an S3 connection using tS3Connection. Then, I'm using tS3List to list all of the objects in a specific bucket. The tS3List component returns exactly 43,000 objects. But, based on the list of objects returned, I believe it's cutting off the results. When I look at the actual data returned, I see that all of the objects in the list are coming back in alphabetical order and the object list does indeed seem to cut off with objects starting with a W. Although I can't get an exact count, I believe this bucket has between 43,000 and 44,000 objects. I know that Amazon's S3 APIs return 1000 objects per set and the tS3List component is able to page through each set of results. However, I think what's happening is that it's not returning the last page of objects, which contains less than 1000 objects.
Is this a known bug, or is there a configuration item that i'm not setting to deal with this?
thanks!

Re: Issue with tS3List component

Anonymous — Wed, 15 Oct 2014 04:43:05 GMT

Hi,
Could you please upload your job setting screenshots into forum which is helpful for us to address your issue.
Best regards
Sabrina

Re: Issue with tS3List component

_AnonymousUser — Wed, 15 Oct 2014 15:17:06 GMT

Sure, here are the screenshots:

Re: Issue with tS3List component

Anonymous — Thu, 16 Oct 2014 05:24:01 GMT

Hi,
Why did you use tRowGenerator component in your job design?
Have you checked component reference:https://help.talend.com/search/all?query=tS3List&content-lang=en with related scenario?
Best regards
Sabrina

Re: Issue with tS3List component

_AnonymousUser — Thu, 16 Oct 2014 15:10:18 GMT

This job is just a test job, to see how these S3 related components work. I had previously been using some components that are on Talend Exchange. So, I wasn't too concerned about using tRowGenerator vs tIterateToFlow at this time.
However, here's a screenshot of the same job using tIterateToFlow. I have the same issue, the tS3List component is returning exactly 43,000 objects. I can also say that we've added files to this bucket in the last few days, yet the tS3List component has been consistent in returning exactly 43,000 rows. So, that goes back to my original question. Is the tS3List component not returning the last page of results, if it's less than 1,000 total objects?

Re: Issue with tS3List component

_AnonymousUser — Mon, 20 Oct 2014 14:57:22 GMT

So, anyone have any updates or thoughts? Is this a real bug in the component that requires a JIRA ticket to resolve?

Re: Issue with tS3List component

Anonymous — Tue, 21 Oct 2014 08:25:20 GMT

Hi,
Could you please open a workitem issue of DI project on Talend Bug Tracker? Our developer will check it to see if it is a bug.
Best regards
Sabrina

Re: Issue with tS3List component

_AnonymousUser — Wed, 22 Oct 2014 19:15:40 GMT

I've created TDI-30918 in JIRA to track this issue.

Re: Issue with tS3List component

Anonymous — Tue, 21 Apr 2015 18:23:08 GMT

Has anyone resolved this? It appears that Amazon S3 limits the "GET" operation of a bucket to 1,000. It's unclear (at least in the link below) if the "max-keys" parameter can be set to an amount greater than 1,000. Another option would be to set a "marker" which would start the next search at the 1,001 object in alphabetical order.
Amazon List Size Limit