topic Re: Qlik Web Connector extremely slow in Connectivity & Data Prep

Qlik Web Connector extremely slow

Narges — Wed, 02 Nov 2022 07:23:35 GMT

Our Qlik Web Connector to June2022 is extremely slow. it took 14 minutes to download a file 24M.

For 100 files between 4kb to 10 kb it took more than half an hour. Is the slow behavior normal for Qlik Web Connector?

Re: Qlik Web Connector extremely slow

Jay_Brown — Wed, 02 Nov 2022 11:23:48 GMT

Hello @Narges thanks for posting!

The speed you are describing does not sound typical. Which actual connector are you using and what type of data source?

If you try other retrieval methods from this same server, such as ODBC, do you see similar speeds?

Thank you!

Jay

Re: Qlik Web Connector extremely slow

Narges — Wed, 02 Nov 2022 11:50:53 GMT

Hi Jay,

We are mostly using sftp connector for downloading data from sftp servers somewhere on internet.

We have not used any other methods as there is no other option for sftp data download to Qlik Sensene except Qlik Web connector

Re: Qlik Web Connector extremely slow

Jay_Brown — Wed, 02 Nov 2022 12:03:00 GMT

Awesome, that helps me understand. SFTP speeds are at the mercy of network speed as well as the speed/load of the FTP server. If you execute this download outside of Qlik, what download times are you seeing?

Re: Qlik Web Connector extremely slow

Narges — Wed, 02 Nov 2022 12:04:33 GMT

Hi Jay, When I use FileZilla or WinSCP, the I can download all data in less than 2-3 min

Re: Qlik Web Connector extremely slow

Jay_Brown — Wed, 02 Nov 2022 12:30:51 GMT

Okay that's interesting, are you using SFTPGetRawFile or SFTPDownload?:

SFTPGetRawFile - Downloads a named file from your SFTP server and returns the results directly. You can use the URL generated as a Web File source.
SFTPDownload - Downloads a named file, or files if you use a wildcard '*', from your SFTP server to a local folder. The table returned shows the number of files downloaded.

I suspect you are using SFTPGetRawFile, and if so, can you try SFTPDownload (or vice versa) and see if the speed is the same?

Also, if you are doing this from a server, can you try running WebConnectors as a standalone from a client machine through the .exe?

Re: Qlik Web Connector extremely slow

Narges — Wed, 02 Nov 2022 14:02:06 GMT

I am using only SFTPDownload

http://localhost:5555/data?connectorID=FileTransferConnector&table=SFTPDownload&

Hi Jay,

I am only using SFTPDownload . This is done via application script to download the data via Qlik web connector and write it on the disk.

A reload task has been defined on the application to reload the app a couple of times during day to download and write data on disk.

We have three dedicated environments, Development with very high hardware specification, and two others with high spec. The same app in all three environments acting exactly the same, and each environment has its own Qlik Web connector.
I can also give it a try via client machine.

I can try

Re: Qlik Web Connector extremely slow

Narges — Wed, 23 Nov 2022 07:21:25 GMT

Hi Jay

I have tested in so many different way.

1.16 G of data, downloaded via Qlik Web Connector in 23 min and 45 S

Via FileZilla 3 min

Re: Qlik Web Connector extremely slow

Jay_Brown — Tue, 29 Nov 2022 16:27:15 GMT

Hi @Narges ,

I've tested this as well. Using the file 1900-1992 City Finance [<2.14 GB] from the US Census. This downloads in 5m 56s inside Qlik Web Connectors, and when put into a Qlik Sense Enterprise load script it downloads in 00:06:36. (see attached)

I am using Qlik Web Connectors June 2022 Patch 2 but there should not be much difference there.

There must be some differences in the way things are happening between the SFTP clients and the servers. Maybe try installing FileZilla on the Web Connectors server and run a similar test?