Data Gateway - File (via Direct Access gateway) – slow, bad performance

[piotrek_w]

Hi All,

We would like to use File (via Direct Access gateway) however it seems to be significantly slower than fetching data from QVD file that resides on our Tenant.

For 2GB QVD file we need 3 to 4 more time to fetch the same file – in scenario when only one file is fetching a the same time. In the case we have 2 files loading in parallel then we need up to 5 times more to fetch this file.

According to Network Activity in Resource monitor the send speed for ConnectorAgent.exe is about 10-12 M B/sec – roughly 80 mpbs.

Questions

1. Do you have similar experience? Is it normal? Can someone provide some statistics from other environments?
2. Does the gateway have any shaping or maximum transfer speeds?
3. Is there anything that we can change in our current set-up to boost the transfer speed?
4. The connectoragent.exe gets an internal IPV6 address for the job, is this ok? Or does it require/prefer ipv4?

Test environment set-up.

• Qlik Gateway host server: Intel®Xenon® Gold 6140 CPU @ 2.30GHz, 512 GB RAM - located in Sweden
• Qlik Tenant: eu.qlikcloud.com
• Test QVD file: Transactions_9; approx. 2GB
• The connectoragent.exe gets an internal IPV6 address for the job
• Microsoft .NET Framework 4.7
• Microsoft .NET Runtime – 8.0.6
• Gateway version 1.7.1
• According to our local IT there is no WPAD – they used winhttp show proxy command to check it:

Test Results:

1.Uploading QVD file directly to Qlik cloud (from gateway host server) – web browser: Edge:

Send B/sec ~ 26M

2. Load test QVD file that was uploaded and saved previously on our tenant – we needed 60 sec to load file:

3. Uploading the same QVD file via Data Gatway.

Send B/sec ~ 10M – we need more than 200 sec to upload file:

4. Uploading 2 files in parallel: 

I created a copy of the test QVD (Transactions_10) file and run two different scripts to check what happen when we would like to read this data in parallel

When I run these scripts almost in the same time I can see only one ConnectorAgent sending data ~ 11 M send b/sec:

When I run second script 10 sec later I can see that we have to ConnectorAgents but the total speed is similar like in the first case...

Thanks in advance!

BR,

Piotr

[marksouzacosta]

Hi @piotrek_w,

There is no way for Data Gateway to be faster than Qlik Cloud - or even close to it.
I know Qlik Data Gateway allows us to access such large files, but I don't think this is a good strategy, especially for Capacity subscriptions - not sure if this is your case.

I would recommend building/storing the QVDs on Qlik Cloud or S3 buckets, however, I can try these two things to improve the performance of Qlik Data Gateway:

1. If possible, consider using Parquet files (BROTLI compression). These files are much smaller than QVD files and load way faster.
2. You can play with your Data Gateway config file increasing the level of isolation for each one of your connectors. Look for Turning on process isolation or changing the number or concurrent reloads on this link for more information: https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/Gateways/direct-access-g... 

This is the configuration settings we use on version 1.7.0 for your reference:

#
# environment variables needed to start the service (ProgramData)
# NOTE: DO NOT LEAVE BLANK LINES
# Below parameters are related to on-premise Direct Gateway
#
# Connector Agent
AGENT_PORT=5050
AGENT_LOG_LEVEL=DEBUG
AGENT_HEALTH_FAIL_MINUTES_LIMIT=5
# DcaaS
DCAAS_PORT=9027
DCAAS_LOG_LEVEL=info
# ODBC Connector
ODBC_DATA_PORT_RANGE_START=50260
ODBC_PORT=3005
ODBC_LOG_LEVEL=INFO
#MAX PROCESS COUNT between 2 and 100
ODBC_MAX_PROCESS_COUNT=60
#None, Load
ODBC_PROCESS_ISOLATION_MODE=Load
# SAP BW Connector
SAPBW_DATA_PORT_RANGE_START=50270
SAPBW_PORT=3015
SAPBW_LOG_LEVEL=INFO
#MAX PROCESS COUNT between 2 and 100
SAPBW_MAX_PROCESS_COUNT=30
#None, Load
SAPBW_PROCESS_ISOLATION_MODE=Load
# SAP SQL Connector
SAPSQL_DATA_PORT_RANGE_START=50280
SAPSQL_PORT=3025
SAPSQL_LOG_LEVEL=INFO
#MAX PROCESS COUNT between 2 and 100
SAPSQL_MAX_PROCESS_COUNT=30
#None, Load
SAPSQL_PROCESS_ISOLATION_MODE=Load
# SAP Connector Package
SAPPACKAGE_DATA_PORT_RANGE_START=50290
SAPPACKAGE_PORT=3035
SAPPACKAGE_LOG_LEVEL=INFO
#MAX PROCESS COUNT between 2 and 100
SAPPACKAGE_MAX_PROCESS_COUNT=30
#None, Load
SAPPACKAGE_PROCESS_ISOLATION_MODE=Load
#SessionLifeTime before ReloadStart
EXTEND_FIRST_REQUEST_TIMEOUT=10
#This property should be used in conjunction with ODBC connector process isolation
ODBC_RELOAD_SESSION_LIFE=60
#Chunks cache memory for a reload
RELOAD_CACHE_MEMORY_MB=100
# File Connector
FILE_LOG_LEVEL=INFO
FILE_DATA_PORT_RANGE_START=50300
FILE_PORT=3045
#None, Load
FILE_PROCESS_ISOLATION_MODE=Load
#MAX PROCESS COUNT between 2 and 10
FILE_MAX_PROCESS_COUNT=10

Regards,

Mark Costa

[korsikov]

I have the same problem. 
Direct upload QVD file to DafaFiles  using browser from machine with Qlik Data Gateway   ~ 200Mb/Sec
Upload using  Qlik Data Gateway  ~ 100Mb/Sec 

How to speedup load using Qlik Data Gateway?

Thanks in advance!