How do incremental refresh QVD's will work with QlikVIew upgrade and migration

[rookianil]

Dear experts,

We are in the process of upgrading QlikView from 11.2 to 12.3 and migrating to new servers at the same time. 

As part of this, we have acquired new servers and a file share then copying all the data from old server to new server but the main issue is we have over 5TB of data includes all qvw, qvd files, logs etc and copy process is taking over few weeks. Could you please help answer below questions.

1) Do we need to copy all qvd files or just qvw files copying will work?

2) Assuming if we need qvd files too, let's say my copy process (from old share to new share) will run two weeks but by then if there are qvws/qvds that gets refreshed on an incremental basis, what happens to those apps when I do my cut over new servers? Do those apps miss data from two weeks? How this can be resolved.

Any help greatly appreciated. Please advise.

[eliran]

Hi,

The question you should ask yourself, do you need the QVD files? do they hold historic data? can't you just re-create them on the new server?

Sometimes it's easier to recreate data, then transfer it to new servers.

If you can't re-create the qvd files, then one approach I would consider is splitting the data.

For example, let's say you have sales.qvd file.

• Sales_History.qvd - all the data until 16.09.2020
• Sales_Incremental.qvd - all the data starting from 16.09.2020

This way, you can transfer the historic data, and then add the incremental load after you finish the migration.

You will need to alter the script at the end, so it will take the incremental file first, and concatenating it with the history - load * from history where not exists(key_to_sales).

I hope my suggestion is clear, let me know.

Eliran.

[marcus_sommer · Accepted solution]

I think you do not need everything within the new environment, for example the log-files. Many of the new log-files will have changes within the data-structure and also the new release is in many ways different so that it's not mandatory to transfer them (at least not immediately else you may just backup them anywhere).

Like already mentioned here all qvd's which could be re-created from the data don't need to be transferred. Where it's not possible you could zip them - depending of the data within the qvd's the compression-rate is between simple text-files which have often a very high rate and pictures which are seldom reduced significantly. I think it's a shot worth. The same is true for all your flat-file data.

The qvw's could be stored without data and contain then just a few KB/MB for the UI.

So in the end the size of the initial transferring data might be much smaller. After it you may need to synchronize some data between the old and the new environment for some time. For such tasks I found robocopy very useful which is really able to synchronize folders so that only changed data are considered.

- Marcus

[rookianil]

Thank you Marcus. This really helps.

We are using robocopy to copy data from old share to new share. What we have decided is to copy whatever files that has modification date greater than 2018 (two years) for all files (qvw and qvd). Do you see any issue with this?

Please advise

[marcus_sommer]

In general this should work. Personally I did such things already several times and I always tried to avoid to slice the task into many parts because it increased the administratively efforts to transfer everything without any mistakes.

Therefore I started as early as possible by creating an extra backup (just to be sure if anything goes wrong), cleaning & preparing data, applications, log-files and so on (removing outdated stuff, zipping, ...) and copying everything within a few days overnight and afterwards I used several robocopy batches to keep everything needed synchron (usually only the data level which is mostly on a yearmonth-level because the qvd/qvw are updated within the new environment - qvd/qvw only if I changed something).

- Marcus

[rookianil]

Thanks Mark. I am asking one more related question please suggest if this not the right forum because I couldn't find any relevant threds.

We also have Qlik Nprinting server running Apps from old Qlik servers. Now since we are moving everything to new Qlik servers and new file share for data, what changes needs to be done at Nprinting  server level? Do we need to update all connections in Nprinting and install QV desktop 12.3 on Nprinting server?

Please advise 

[marcus_sommer]

Unfortunately we didn't invest in NPrinting yet and therefore I have no experience with it.

It's not directly related to your question and may be already too late for your migration process but I would split the task of moving to a new environment and the upgrade of QlikView.

My preferable way is at first to mirror the existing production environment within the new environment and if everything worked like expected I will go on with the upgrade. This means not lesser work but it reduced the scope of possible errors (proxies, firewall, access rights (storage, ports, user, connectors, ...), settings, configurations and so on) could be excluded from issues which may come from the new release). Beside this it refreshed the knowledge how the old environment is configured so that's easier to enable everything correctly within the new release.

- Marcus

[rookianil]

Thanks Marcus. You are right, now at this point I can't go back to just migrating. We wanted to migrate to new server first but since we are working on moving everything to new server, thought of doing upgrade also same time but you are right things would have been easier if we were just doing migration only. But with 11.2 is ending support  soon we thought of upgrade also same time.

[rookianil]

Hi Marcus,

one last question, I have installed Qlikview 12.3 on new servers and pointed them to new file share and updated QVPR data. But, my data copy from old share to new share is not completed yet (.qvw/.qvd). Can I start testing applications in new system with whatever that is already being copied to new share while data is still copying over or do you suggest to wait until all data is copied?

[rookianil]

Thanks Eliran. We have thousands of apps and I am not sure whether scripts have this mechanism of building QVDs from scratch if one doesn't exist . That is why I wanted to copy everything to new file share but issue we are facing is this is running for almost a month now. Can I start validating whatever already copied while the process is still running to copy?