topic tS3List and filename in Talend Studio

tS3List and filename

Anonymous — Fri, 30 Jun 2017 08:29:25 GMT

Hello,

I'm using the tS3List component to list files in a bucket according to a prefix key. At each iteration, it gives me :

tS3List_1_CURRENT_BUCKET : the current bucket
tS3List_1_CURRENT_KEY : the global current key path

When using the tS3Get component, it asked me for a the destination filepath, which I want to be dynamic according to the iteration. Why the tS3List component doesn't give us directly the 3 following paramaters ?

current_file
current_file_path
current_directory

Re: tS3List and filename

Anonymous — Fri, 30 Jun 2017 08:38:38 GMT

This is because S3 is not a normal drive with directory and file path. It is a key/value pair. The value being any byte stream object. As per the example from AWS documentation, an object in S3 has a key, not a filename, filepath or directory.

Re: tS3List and filename

Anonymous — Wed, 19 Jul 2017 13:07:42 GMT

I am interested in filename from tS3List component, but there is no global variable available. I have used StringHandling to do it, but not very happy with it.

The files are stored in S3 bucket as abc/abc/abc/abc/filename.txt, as I know this I have used StringHandling.Right, StringHandling.LEN and SQLike.mid functions.

StringHandling.RIGHT( ((String)globalMap.get("tS3List_1_CURRENT_KEY")),(StringHandling.LEN(((String)globalMap.get("tS3List_1_CURRENT_KEY"))) - StringHandling.LEN((String)SQLike.mid_index(((String)globalMap.get("tS3List_1_CURRENT_KEY")),"/",4)))-1)

Is there a way to parse tS3List_1_CURRENT_KEY upto first "/" from right? That should give me filename little more efficiently and I do not have to hack with values such as '4'

Re: tS3List and filename

Anonymous — Wed, 19 Jul 2017 13:17:57 GMT

I use the following code to get the filename :

((String)globalMap.get("tS3List_1_CURRENT_KEY")).substring( ((String)globalMap.get("tS3List_1_CURRENT_KEY")).lastIndexOf("/") + 1 )