qlik data transfer

[hafaneh]

Hello
Im using qlik cloud but i take the data from the server(qlik on premise) 

,i load the data to the qvd then upload it daily on One drive,

i wanna ask if i use the qlik data transfer will it be more efficient and faster to upload it daily to the cloud from the qlik on premise? cuz my qvd is getting bigger now i need a faster way to upload it.  

[Miguel_Angel_Baeyens]

Qlik Data Transfer can send files to your Qlik Cloud tenant, however it will not copy files on OneDrive or any other storage, if you need that you will have to use some other software to copy from your servers to OneDrive.

Note that Qlik Data Transfer requires a dedicated server, that is, you should not install it along with a QlikView or Qlik Sense server.

https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/QlikDataTransfer/qlik-da...

If QVDs getting bigger is the main problem, I would suggest revisiting the loading script to store smaller QVDs anyway, for example, if you are storing all the data for one year into one single QVD, store instead data for 1 month or 1 week. You will have more files but they would be smaller and faster to transfer.

[Miguel_Angel_Baeyens]

Also, Qlik Data Transfer will use the same connection that your servers have. If that connection is slow, or the upload link is smaller, or the files are very big, or all of them, you can expect some time until the file is sent across.

I would also explore mounting your OneDrive into the on-prem server (for example this: https://support.microsoft.com/en-gb/office/sync-files-with-onedrive-in-windows-615391c4-2bd3-4aae-a4... but check with your Microsoft / IT team) to store directly in the cloud so your Qlik Cloud tenant would read from that same location using the OneDrive connector: https://help.qlik.com/en-US/cloud-services/Subsystems/Web_Storage_Provider_Connectors_help/Content/C....

[HAmzehAF]

hello @Miguel_Angel_Baeyens , 
thank you for the reply.

i wanna ask you about the how can i store more qvds for the same qvd how can i link them together, 

my qvd now is 5gb it has 2022 data for sales and 2023 data for sales ,
im thinking about keeping the 2022 on the cloud and use binary to link the two toghther  but we need more than 5gb per app to do it , 

but if can tell me the way on how i can do the smaller qvds and link them together that would be so helpful \

,

and as you know cloud apps have 5gb per app if i used smaller qvd would it help with sizing for the app 5gb size? 

[Miguel_Angel_Baeyens]

You can use a variety of techniques to incrementally load data, take a look at the examples in the Help site here: https://help.qlik.com/en-US/sense/November2023/Subsystems/Hub/Content/Sense_Hub/LoadData/use-QVD-fil.... For Qlik Cloud is pretty much the same thing: https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/LoadData/use-QVD-files-i....

I would address the files that you have today to reduce that 5Gb by creating an empty Qlik Sense app to read the QVD and break it down into smaller QVDs, for example, per month.

The summary of steps would be:

1. Identify the month field or create one based on a date in your 5GB QVD
2. Loop through all the months to load each month from that QVD and store each month into a separate QVD

Moving forward, every new month you have to load, you store it into a new QVD, so instead of 2 big files for years 2022 and 2023, you will have 24 smaller QVD files for each month.