Skip to main content
Announcements
Join us at Qlik Connect for 3 magical days of learning, networking,and inspiration! REGISTER TODAY and save!
cancel
Showing results for 
Search instead for 
Did you mean: 
herard_bertrand
Partner - Contributor III
Partner - Contributor III

How to load data from a S3 partitioned database ?

Hi,

I've tested amazon S3 connector to load a simple parquet file in Qlik Cloud (Qlik Sense) and it works.

Now, is it possible to natively and quickly load a bunch of partitioned files ? for example, if my table is partioned by date :  

myS3Bucket/myS3Folders/mytable

is partitioned with a lot of parquet files such as :

myS3Bucket/myS3Folders/mytable/data=2024-01/part-lotofnumbers.snappy.parquet
...
myS3Bucket/myS3Folders/mytable/data=2023-12/part-lotofnumbers.snappy.parquet

 

1) Is there a way to use the S3 connector to load directly every needed file but just indicating myS3Bucket/myS3Folders/mytable ?

2) if not, is there any tip to handle it easily in the script ? (for example, I tried to use * and ? character as joker but theses jokers doesn't works)

 

(note : I know it'is possible to use the S3 Metadata connector to get all objects name, then select files in a manual loop, but it not really easy to write and use).

 

Thanks

0 Replies