*** SORRY*** This was my fault, I had another error in the job that was making it look like the tS3List was misbehaving, when it was actually not.
I want to list all the files in all subdirectories of a particular S3 key. I set up a tS3List component, gave it the bucket and key, and it just listed the subdirectories, and not the files within them.
So I added a second tS3List to take the results of the first, and use the directory returned as the key. This seemed ok, except that it was duplicating the entries - sometiumes they were showing up twice, sometimes three times.
In order to debug it, I temporarily removed the second tS3List, and now it is returning all the files in the subdirectories! I have not changed anything in the first tS3List, there is so little in there to change that I can be pretty confident of that.
Another quirk is that it is not listing all of the directories themselves - it lists all the files in all the subdirectories, but only some of the subdirectories show up as an entry of their own.
Any suggestions why the tS3List might behave erratically like this? I am reluctant to put this into production with this uncertainty hanging over its behaviour.
Sorry, I found my problem, and it was nothing to do with the S3List component. I was discarding the data at a subsequent step due to a faulty regular expression and hadn't noticed.
Could you please share the screenshot of the job flow where you are using tS3List and could you please also share the component screenshot of the Ts3List you have used for analysis?
While posting, please remember to blur the S3 access information.
Sorry, I found my problem, and it was nothing to do with the S3List component. I was discarding the data at a subsequent step due to a faulty regular expression and hadn't noticed.
