Qlik Data Transfer product Lifecycle - End of support

[RenzoFantini]

Good morning community/experts,
In July, we migrated our QlikSense On-Prem to the Cloud (premium—capacity based—50GB—contract) and started using Qlik Data Transfer (Release Nov 2022—Version 10.1.0—End of Support 02—May—2025) to move our on-prem data to the cloud.
Since we saw that the end of support of Qlik Data Transfer is set for May 2025 I would like to know if a new release will be released soon.
If not, what is the best replacement? Qlik Data Movement? Qlik Data Gateway - Direct Access (since we have a "premium - capacity based - 50GB - contract" we would like to avoid impact on capacity).

Many thanks,

[BuildItStrong]

We replaced our Qlik Data Transfer with Qlik Data Gateway.  I liked Data Gateway better.

[BuildItStrong]

I realize that my response is rather vague.  Here are the things that I remember when I think about data gateway vs data transfer.

First, we run most of our enterprise on virtual servers.  We ran both data transfer and data gateway from virtual servers.  The network and machine configurations were identical for us (same memory and processor cores).  I'm not saying that it will be the same for you.   It was for us.

Second, as I recall the connection from data transfer to a database was controlled by the service account that ran the data transfer application on your network.  Any user that could gain access to data transfer could see all of the tables, etc. accessible to the user account that owned the services of data transfer.  This did not provide much governance.  We were very restrictive of the users that could actually connect using data transfer. 

Data gateway has a better governance model.  First, you can control who has access to each data gateway by associating the data gateway to a Qlik Sense space, and then assigning users to that space.  So you can have tight control over the users that can reach the databases accessed by each data gateway.  You can have several data gateways.  Further, users can pass their credentials through the data gateway to gain access to the databases served by that gateway.  This lets us further control what each user sees when the user connects to a database engine through the data gateway.  If a user has read privileges on the database server to a restricted subset of databases or tables, when the user connects to the database through data gateway with the user's own credentials, then the security governance of the database server still controls what the user can see.   So, the governance restrictions of data gateway are far more capable than that of data transfer.

Otherwise, the data transfer speed is about the same.  Hardware and software requirements area about the same.  Software version and update requirements are easily accessible through the data gateway portion of Qlik Sense Cloud admin, as is the update software for that portion of data gateway that runs inside of your firewall.  Software updates are easy - almost trivial.

The system stability of Data Gateway, if you keep the software versions somewhere near up-to-date, is very good.

[RenzoFantini]

Good morning and many thanks for your detailed answer.

• More or less our configurations is similar to your configuration: we have both of them installed on Virtual Machine with same characteristics.
• I also prefer data gateway direct access but, since our contract is based on capacity, we use direct access only to load small piece of data to avoid big impact on space consumption.
  When you use direct access you pay double: You pay for what you read (eg data ingested by a sql query) + you pay for what you write (eg QVDs)

Anyway my real point was to understand if Qlik will extend Data Transfer support and, if not, what is the best alternative way to move data from on-prem to cloud reducing impact on capacity? Could be Qlik Data Gateway Data Movement (we already tested it and it seems good)?

Have a nice day,

[BuildItStrong]

I certainly can't speak for Qlik about what they might do with long-term Data Transfer support.  I can share what we do to minimize our overall Qlik bandwidth use.

First, we have a data lake inside of our firewall, in our internal network.  We do this because we want a single source of truth for all of our reporting and data needs, and we have a considerable collection of legacy reports.  If a report needs to be accurate as of this moment, we try to write it against the transactional system that contains the needed data.  If the need is more analytical, we use the data lake.

Our data lake is inside our firewall because that works best for us given our particular business needs.  It could certainly be on a cloud resource somewhere.

We also use our data lake for governance purposes, for master data management and other purposes.

We started our journey to Qlik Cloud by mirroring large blocks of our data lake in Qlik Cloud as QVD files.  We developed a collection of Spaces in Qlik Cloud to hold our data.  We mirrored our data access model in our data lake in theses new Qlik Cloud spaces, and used Active Directory (or whatever Microsoft chooses to call that today) to control access to these spaces in the same way we use AD to control access in our data lake.

We then loaded a big chunks of our data lake into QVD files in Qlik Cloud into these access controlled spaces.  Our citizen developers used this data to develop new dashboards in Qlik Cloud.

We did this in part because Data Transfer did not let us control who had access to data when directly connecting to our data lake.  This approach let us give access to data while controlling that access in a way that mirrored our data lake.

We ended up with a large collection of QVD files, and a significant fraction of our data lake mirrored into QVD files in Qlik Cloud....   With plans of adding more.

Data Gateway lets us change that model.  Data Gateway lets us control user access to the data lake from Qlik Cloud.  Users connect to the data lake from Qlik Cloud with their own credentials, and they can only see the data that has been made available to them in the data lake.

We are systematically removing QVD files from our Qlik Cloud collection, and replacing the load scripts of dashboards with Data Gateway load scripts. We started our journey to Qlik Cloud by mirroring large blocks of our data lake in Qlik Cloud as QVD files.  Now we are slowly retiring that QVD file collection.  We find that our total data load bandwidth is smaller.

There are compromises to be made.  Some of our QVD files are used by a number of dashboards.  It is more cost effective and performant to leave those in our QVD file collection.  This is not a perfect solution.  It can cause confusion in our user base about where to find data.  We are working to minimize this with a data catalog.

There are other advantages.  We are continually pushing to keep our data lake more and more current.  Originally, the bulk of our data lake was loaded once per day.  We are moving towards a near-real time data lake.  Loading dashboards directly from our data lake helps for those few dashboards that need data loads more than once per day.  

More current data also makes Qlik Alerts more useful, but we are only starting to explore those on an enterprise level.

These are my thoughts.