How to load data from a S3 partitioned database ?

herard_bertrand — Wed, 19 Mar 2025 22:43:16 GMT

Hi,

I've tested amazon S3 connector to load a simple parquet file in Qlik Cloud (Qlik Sense) and it works.

Now, is it possible to natively and quickly load a bunch of partitioned files ? for example, if my table is partioned by date :

myS3Bucket/myS3Folders/mytable

is partitioned with a lot of parquet files such as :

myS3Bucket/myS3Folders/mytable/data=2024-01/part-lotofnumbers.snappy.parquet
...
myS3Bucket/myS3Folders/mytable/data=2023-12/part-lotofnumbers.snappy.parquet

1) Is there a way to use the S3 connector to load directly every needed file but just indicating myS3Bucket/myS3Folders/mytable ?

2) if not, is there any tip to handle it easily in the script ? (for example, I tried to use * and ? character as joker but theses jokers doesn't works)

(note : I know it'is possible to use the S3 Metadata connector to get all objects name, then select files in a manual loop, but it not really easy to write and use).

Thanks

topic How to load data from a S3 partitioned database ? in Data Movement & Streaming

How to load data from a S3 partitioned database ?