Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Qlik Open Lakehouse is Now Generally Available! Discover the key highlights and partner resources here.
cancel
Showing results for 
Search instead for 
Did you mean: 
herard_bertrand
Partner - Creator
Partner - Creator

How to load data from a S3 partitioned database ?

Hi,

I've tested amazon S3 connector to load a simple parquet file in Qlik Cloud (Qlik Sense) and it works.

Now, is it possible to natively and quickly load a bunch of partitioned files ? for example, if my table is partioned by date :  

myS3Bucket/myS3Folders/mytable

is partitioned with a lot of parquet files such as :

myS3Bucket/myS3Folders/mytable/data=2024-01/part-lotofnumbers.snappy.parquet
...
myS3Bucket/myS3Folders/mytable/data=2023-12/part-lotofnumbers.snappy.parquet

 

1) Is there a way to use the S3 connector to load directly every needed file but just indicating myS3Bucket/myS3Folders/mytable ?

2) if not, is there any tip to handle it easily in the script ? (for example, I tried to use * and ? character as joker but theses jokers doesn't works)

 

(note : I know it'is possible to use the S3 Metadata connector to get all objects name, then select files in a manual loop, but it not really easy to write and use).

 

Thanks

0 Replies