Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements
Qlik Open Lakehouse is Now Generally Available! Discover the key highlights and partner resources here.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable
‎2017-07-17 10:22 AM

AWS S3 file properties

Hello,

 

I would like to know if there is a way to get file properties (like size, timestamp) of a file places in S3 bucket. 

 

I have tried the following but unable to get file size and timestamps. Also does this differ if the file in S3 bucket is .gz file.

 tS3 Connection

tS3FileList --iterate--> tFIleProperties --iterate--> tIterateToFlow --main--> tLogRow

Thanks,

A

Labels (2)
Labels
979 Views
0 Likes
1 Solution

Accepted Solutions
PhilHibbs
Creator II
Creator II
‎2020-02-04 12:00 PM

SOLVED!

 

If your tS3List component is called, for example, tS3List_1, simply add the following entry to a Date column in the subsequent tIterateToFlow:

 

objectSummary_tS3List_1.getLastModified()

 

This is relying on undocumented internal variables. I have tested it in Talend v6.3 and 7.1. Another drawback is that if you copy and paste a pair of components, the globalMap.get("tS3List_1_CURRENT_KEY") call will be updated with the new component name, but the internal variable objectSummary_tS3List_1 will not be.

View solution in original post

979 Views
0 Likes
11 Replies
Anonymous
Not applicable
‎2017-07-18 05:11 AM
Author

Hello,

So far, you have to retrieve your files from S3 and keep them on local firstly.

The work flow should be: tS3Connection-->tS3Get-->tFileProperties

Best regards

Sabrina

 
797 Views
0 Likes
Anonymous
Not applicable
‎2017-07-18 05:35 AM
Author

Thanks Sabrina for your reply. If that's the only way then I'll take it..

 

797 Views
0 Likes
Anonymous
Not applicable
‎2017-07-24 03:52 AM
Author

As we have a client restriction not to download the file locally,canu please let us knw if there is a way other than downloading? or download/read file properties and delete in one component soo that we cant eve break this flow .ultimatley we shd nt have any control on seeing the file content.

797 Views
0 Likes
Sid3
Contributor III
Contributor III
‎2017-08-25 09:35 AM

Hi Sabrina,

tS3Connection-->tS3Get-->tFileProperties

In this approach once file gets downloaded(using ts3Get) it will have current time stamp on it. we will not get the actual time stamp which it has in S3 bucket.

how to get the time stamp of file without downloading from server?

Any inputs?

Thanks,
Sid
797 Views
0 Likes
Sid3
Contributor III
Contributor III
‎2017-08-28 06:12 AM

Hi Folks,

 

Any suggestions here?

 

Thanks,

Sid

797 Views
0 Likes
Sid3
Contributor III
Contributor III
‎2017-08-29 09:54 AM

Hi Guys,

 

I got the work around for this issue. It's using S3 Command line.

 

PFA job,component and console output images.

 

Thanks,

Sid

 

 


connsle_output.JPG
tjava_comp_properties.JPG
tsystem_comp_properties.JPG
S3Filepropertiesjob.JPG
797 Views
0 Likes
Anonymous
Not applicable
‎2018-03-20 05:08 PM
Author

Hi Sid

I tried to follow your approach, but I get an error...see the thread : https://community.talend.com/t5/Design-and-Development/unable-to-execute-command-using-tSystem-compo...

 

 

797 Views
0 Likes
Anonymous
Not applicable
‎2018-08-14 06:42 AM
Author

Hi,

 

I have got same constraints. Cannot download the file locally. 

 

We can use AWS CLI however then go off from using the components. As a suggestion can we have something in S3Get or a seperate component to read the file properties on S3?

797 Views
0 Likes
PhilHibbs
Creator II
Creator II
‎2020-02-04 07:03 AM

Bumping this thread - has anything changed in the last 2 years? I can't see anything in the v7.0 docs.

 

I want to check to see if the files all have timestamps that are close together, i.e. there are no files left over from a previous day. There are hundreds of files and they are all many gigabytes so I can't download them!

797 Views
0 Likes