Qlik Data movement gateway - "other supported" Linux platforms

[mattdetwiler]

We are looking to move data from Salesforce and our on-prem MS SQL server.  As far as we can tell if we want to store the data in a QVD in Qlik SaaS we will need to do this via a Qlik Data Movement Gateway.

The documentation specifies "Redhat Linux and other compatible platforms" however when we attempted to set this up on Ubuntu the rpm doesn't seem to convert cleanly enough to work/start the service.

Our first question, Do we need a data movement gateway to achieve our objective?

Can we get Ubuntu to work?  It's strange that we need to set up a server to use Qlik SaaS.  (Our main reason to move was to have no local servers).

[marksouzacosta · Accepted solution]

Hi @mattdetwiler,

Qlik Data Gateway - Data Movement is required only for Qlik Cloud Data Integration (QCDI) Connections.

If your idea is just to load data from Salesforce and SQL Server on-prem to Qlik Cloud as QVD you will need:
- For Salesforce: just use the Salesforce Data Connection natively available in Qlik.

- For SQL Server on-prem: Qlik Data Gateway - Direct Access: Qlik Data Gateway - Direct Access overview | Qlik Cloud Help. This one is easier to install than the Data Movement.

Regards,

Mark Costa

[mattdetwiler]

Thanks Mark!   From what I understood that would allow us to load the data but not to move/store it in a QVD.  It's really confusing the difference between the 2 gateways and I thought storing/not storing was the primary difference. 

If you can point me to anything that would explain how to set up what we need it would be a huge help.

[marksouzacosta]

For storing QVD files you don't need the Data Movement, only the Data Gateway.

Data Movement is a component required only by QCDI. It allows us to connect to multiple data sources (Cloud and On-Prem) and move the data to multiple Targets (also on-prem or Cloud, including QVD in Qlik Cloud). QCDI is an option but more complex than regular Qlik Script with Data Gateway.

Data Gateway is a component required only to connect to on-prem Data. It creates a bridge between your Qlik Cloud Tenant and your on-prem server. I would take this option since it is way easier to deal with Data Gateway than Data Movement.

So, for Salesforce, nothing special is needed. Just create a new Qlik Sense Application, go to the Load Editor, add a Salesforce Data Connection. Load the tables you need and store as QVD. A few more information about this process here: Working with QVD files | Qlik Sense on Windows Help

To get data from SQL Server with Data Gateway will be very similar to Salesforce after you have setup the Data Gateway Server.

[mattdetwiler]

Thanks Mark!  I'll check that all out!

[marksouzacosta]

A little more about the SQL Server connection.

On the Create data connection window you will find a Microsoft SQL Server (via Direct Access gateway) option available:

After selecting the Microsoft SQL Server (via Direct Access gateway) option, you will see the list of Data Gateways you have available to connect to your on-prem SQL Server. Fill the server name, etc as you are on an on-prem server:

[mattdetwiler]

Commands seem pretty different in SaaS for using and accessing Qvds.  I think I am able to use the store command but I'm not able to see how/where the storage is located.  Do we have to sign up for an S3 bucket to store QVDs?

[marksouzacosta]

Hi @mattdetwiler,

No, you don't need a S3 Bucket. Qlik stores the QVD internally in its DataFiles area.

I did a short video showing how to create a Qlik Sense Application from scratch to Load Data from a Snowflake connection that I have setup - in your case will be a Salesforce - and store as a QVD file.

Later I show up how can you load the data from a QVD file - usually this is made in another separated Qlik Sense Application.

I hope this helps!

[mattdetwiler]

@marksouzacosta Thanks this is awesome!

For anyone reading here's what I did:

I first tested creating a qvd in my personal space:  (The only thing you need to change here is your qvd name)

      Store TABLE into ‘lib://DataFiles/FILE_NAME.qvd’(qvd);

Then I tested it in a public space that I created:

     Store TABLE into ‘lib://SPACE NAME:DataFiles/FILE_NAME.qvd’(qvd);
     Example:  STORE [Opportunity] INTO 'lib://SFDC_QVD:DataFiles/SFDC_L1_Opportunity.qvd'(qvd);

In this example everything is done from the data load editor.  Opportunity is the table I loaded from my data connection/source.  "SFDC_QVD" is the name of the shared space I created and "SFDC_L1_Opportunity.qvd is the name of the QVD file I created.

If you are creating apps to build qvd files for use in other apps you can create multiple sections named by table names to create each QVD for a data source.