topic Qlik Data Catalog Load data from S3 in Catalog and Lineage

Qlik Data Catalog Load data from S3

vishavb007 — Tue, 07 Dec 2021 22:26:21 GMT

Hi All,

I am able to load data(data from single file) from S3 bucket into Qlik Catalog.

But I need to load the folder from S3 in which we have multiple files , kindly let me know if it is possible to load the folder & if it is yes , please let me know how to do it.

Kindly let me know if any further details are required regarding the use case.

Regards,

Vishav Bhatia

Re: Qlik Data Catalog Load data from S3

Hordy_Beauzile — Tue, 07 Jun 2022 19:16:59 GMT

Hi @vishavb007,

After adding the source connection of the S3 bucket, you should be able to select the folder as a source. After creating the source you should then be able to load the files through the established source connection.

https://help.qlik.com/en-US/catalog/May2022/Content/QlikCatalog/Source/JDBC_New_Source.htm

Best,

Hordy

Re: Qlik Data Catalog Load data from S3

ducetcro — Thu, 25 Jan 2024 12:14:42 GMT

Split the files into 100Mbs chunks and convert to parquet (for best read performance). Partition them by Day or by Hour (to reduce cost when querying). Create a Redshift Spectrum external table pointing to the bucket and access the files directly without needing to move them.