Sense 3.0 migration strategies and experiences

[mountaindude]

I am looking at options for migrating a Sense 2.2.4 environment (one central node and a few rim nodes) to 3.0.

The obvious way is to do an in-place upgrade of the existing 2.2.4 system, but what if something goes wrong... It's a production environment with lots of users. Yes there are backups, but still.

Re backups: In this case the servers are virtualised, and even though we snapshot the disks of the Sense servers each day, for some reason it is not possible to recreate a new server based on such a snapshot.

The idea would be to just spin up a new server created from a disk snapshot of the failed Sense server - but after doing this, the newly created server is not recognised by the Sense cluster. It has the same IP, same certificates etc - but for some reason the new server's services are not recognised as running by the central node. The services (engine, proxy etc) are running on the new server, but not recognised by the central node. It is possible to access all files etc on the new server - so for backup it's fine - but still not a good disaster recovery solution. All in all - it's really scary to upgrade to a new version without a good fallback strategy if the upgrade fails...

I would think others are facing the same challenge, so maybe there are ideas and experiences to be shared here?

A couple of questions come to mind:

1. Have you done an upgrade to Sense 3.0? What are your experiences?

- From what version did you upgrade?

- What does your Sense environment look like (single node vs multiple nodes etc)?

- Did you do a repository database cleanup (as mentioned in the 3.0 release notes) prior to the upgrade?

- Approx size of the Sense environment (e.g. approx number of apps, if that can be shared)

- ... other relevant info that can be shared.

- And most importantly: Did the upgrade work, and did you get a working 3.0 environment as intended?

2. Ideas for alternative migration strategies

- It would be nice if it is possible to "clone" the production Sense 2.2 environment onto a new server, upgrade that central server to 3.0, verify that everything works, and then add additional Sense 3.0 rim nodes as needed.

But is it possible to clone for example a central node onto a new server, and run it there, without it interfering with the existing production environment? I guess the new/cloned central node could be placed in an isolated network... But even so - I guess Sense certificates are created based on server names... which probably means cloning like this won't work...

- Can apps be bulk exported from Sense 2.2, then bulk imported into 3.0?
Apps can be serialised to JSON, but then things like bookmarks and data are not included..  A solution that keep things like bookmarks, custom user sheets etc intact is needed.

- Other possible migration strategies?

[shaun_lombard]

Hi Göran,

We have upgraded our development environment from 2.0.7 to 3.0.

It is a 2 node setup on AWS servers. We have about 110 apps, and we did not run the database clean-up steps.

There were a few issues with the upgrade.

1. The central node stalled while it was in the process of migrating the apps. We could not monitor the progress of the app migrates because the QMC would not respond. We eventually tried restarting the services, which also did not work and we ended up restarting the server. After the server came back we were able to manually migrate the 20 apps that were still in an unknown status.
2. The upgrade of the second node went well but we did observe a long period of time that the server was running at over 95% CPU.
3. The final issue we encountered I believe to be related to our use of a User Directory Connector. For some reason after the services are started it appears the UDC was synchronised, even though we had it disabled. This caused a few of our users to be marked as "removed externally" (including my RootAdmin user). We recovered the users by deleteing them and re-adding them (which orphans all the objects they own). This kept happening until I renamed the UDC user directory to something else.

For the production upgrade we were also planning on snapshotting the virtuals for our backout strategy. We would not try create a new virtual server from the snapshot, but instead replace the existing server with the snapshot. This has not been tested.

Shaun