Skip to main content
Announcements
Qlik Introduces a New Era of Visualization! READ ALL ABOUT IT
cancel
Showing results for 
Search instead for 
Did you mean: 
kanhomcake
Contributor III
Contributor III

Is There a Limitation in The Number of Files You Can See When "Select data" from Azure Blob Connection?

hi Community, 

I'm doing some testing to move from QS on-prem to SaaS. Then I come to encounter an issue. Here is what I've done so far.

- On my tenant, I create a connection to Azure Blob storage.
- In one of the folders in this Azure Blob container, there are approx. 68,000 qvd files.
- I want to Select the data from this container/folder which then the issue occurs, please see the screenshot below.

kanhomcake_0-1671807737944.png
- After this I test loading the data from the file zzzzz.qvd for example and it works
- Then I test with Filelist function e.g. FOR EACH _file in filelist('lib://Admin:AZ_QAS_QVDRepository/GL3/'); and count how many times it loops, again only 10000 counts

Anyone has the same issue? Is this known as a limitation for Azure Storage connector in QSE SaaS? Or limitation of Azure Storage API that QSE SaaS is using?

Any clue/workaround would be helpful 🙂

Thank you so much 🙂
KC

 

1 Solution

Accepted Solutions
Rakesh_HB
Support
Support

Hello @kanhomcake ,

We are interested to see what the UX is like with 68k files! Could you please share it?

And, yes, we have a limit of 10000 and it will be increased in the next release.

Rakesh HB

View solution in original post

6 Replies
kanhomcake
Contributor III
Contributor III
Author

hi again, found this https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs?tabs=azure-ad but the default Maxresults is 5000 🤔

Rakesh_HB
Support
Support

Hello @kanhomcake ,

We are interested to see what the UX is like with 68k files! Could you please share it?

And, yes, we have a limit of 10000 and it will be increased in the next release.

Rakesh HB
kanhomcake
Contributor III
Contributor III
Author

hi @Rakesh_HB ,

Thank you for the confirmation and information 🙂

When we (developers) work with the QS on-prem, we haven't had any performance issue with these files (approx. 500GB in total). The performance is acceptable when we load data from qvd files from this 68k-file-folder using wildcards (e.g. datafile_??????.qvd). When we move to Cloud and store the same files on Azure blob, we haven't experienced any bad performance either managing the files via Azure Web Portal or Storage Explorer.

When trying to access those files via QSE SaaS, I can't judge the performance because as you know, we can see only the first 10000 files 🙂 I'll for sure like to share with you again when the limit is increased 🙂

Many thanks again for your reply 🙂

Best regards, 
KC

Rakesh_HB
Support
Support

Hello @kanhomcake ,

I confirm that the limit has now been removed. Please check and confirm from your end.

Rakesh HB
kanhomcake
Contributor III
Contributor III
Author

hi @Rakesh_HB , no, not really, I can still see only 10k file from select data. Also, when I loop for each file, I got only 10k-record count.

Rakesh_HB
Support
Support

Can you please check by creating a new connection and testing it?

Rakesh HB