QS S3 Connection for Non Public files

[Anonymous]

Hi All,

I cannot currently connect to a private S3 bucket using the QS connector, as this requires that the buckets be made public.

Is there any way I can overcome this with the connector or indeed a web file / REST connection, without having to compromise on file security?

Thanks

[Levi_Turner]

Either by using the S3 connector in the Qlik Web Connectors OR a complicated method referenced above.

The schema of the hard method would be:

• Setup a script to compute the exact values used in the variables.
  • e.g. node s3header-authorization.js outputs LET vAuthorization='xxxx-xxx-xxxxxx Credential=xxxxxxxxxxxxxxxx/20180130/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'; which is then stored in a .TXT or .QVS file. Repeat for all headers.
• In the load script use include files to pull in these variables
  • e.g. $(Must_Include=lib://Scripts\s3header-authorization.qvs);
• Setting up a process to reload those tokens every X minutes.

It certainly could work but the S3 connector seems much easier to implement, in my view.

[Anonymous]

Hi Levi,

QS does not provide the ability to code SHA-256, which the real-time generation of the AWS signature requires - only Hash256 which isn't the same.

The s3 connector provides the error shown.

Example code looks like it would work..(if SHA-256 encryption was indeed possible.)

---------------------------------------------------------------

let date = TEXT(date(today(),'YYYYMMDD'));

let dateIS08601 = TEXT(REPLACE((DATE(now( ) ,'YYYYMMDD')&'T'&DATE(now() ,'hh:mm:ss')&'Z'),':',''));

let canonical_request =
'GET
/

host:axis360.s3.amazonaws.com'&chr(10)&
'x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'&chr(10)&
'x-amz-date:'&'$(dateIS08601)'&chr(10)&chr(10)&
'host;x-amz-content-sha256;x-amz-date'&chr(10)&
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';

let hash_canonical_request = Hash256(text('$(canonical_request)'));

let signing_key = Hash256(Hash256(Hash256(Hash256("AWS4" + "xxxx-xxxx-SHA256","$(date)"),"us-east-1"),"s3"),"aws4_request");

let String_to_Sign =
'AWS4-HMAC-SHA256'&chr(10)&
'$(dateIS08601)'&chr(10)&
'$(date)'&'/us-east-1/s3/aws4_request'&chr(10)&
'$(hash_canonical_request)';

let signature = Hash256('$(signing_key)','$(String_to_Sign)');

let vAuthorization='xxxx-xxxx-SHA256 Credential=xxxxxxxxxxxxxxxx/'&'$(date)'&'/us-east-1/s3/aws4_request,
SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature='&'$(signature)';
let vSha256='e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855';
let vDate='$(dateIS08601)';

let URL_path = 'https://s3.amazonaws.com/publytics-ftp-backup/receipts/PUBLYTIC_RECEIPTS_20170914.TXT';
let vURL_path = replace(URL_path,'"', chr(34)&chr(34));

Thanks for your help so far.

I'm not sure how the s3 connector error can be resolved given that the same details can be passed though using a REST connection or Postman.

Thanks

[Ken_T]

@Levi_Turner can you give some possible causes for the 403 error?  you mentioned it could be a permissions issue. permissions on the amazon side or on the qlik side?   thanks!

[priyankapassi]

Any possible way to fetch txt files instead of CSV?Please help.

Thanks