Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Want to see what's currently in public preview? We've pulled it all together in one place. Check it out here
cancel
Showing results for 
Search instead for 
Did you mean: 
herard_bertrand
Partner - Creator II
Partner - Creator II

How to load data from a S3 partitioned database ?

Hi,

I've tested amazon S3 connector to load a simple parquet file in Qlik Cloud (Qlik Sense) and it works.

Now, is it possible to natively and quickly load a bunch of partitioned files ? for example, if my table is partioned by date :  

myS3Bucket/myS3Folders/mytable

is partitioned with a lot of parquet files such as :

myS3Bucket/myS3Folders/mytable/data=2024-01/part-lotofnumbers.snappy.parquet
...
myS3Bucket/myS3Folders/mytable/data=2023-12/part-lotofnumbers.snappy.parquet

 

1) Is there a way to use the S3 connector to load directly every needed file but just indicating myS3Bucket/myS3Folders/mytable ?

2) if not, is there any tip to handle it easily in the script ? (for example, I tried to use * and ? character as joker but theses jokers doesn't works)

 

(note : I know it'is possible to use the S3 Metadata connector to get all objects name, then select files in a manual loop, but it not really easy to write and use).

 

Thanks

0 Replies