Qlik Community

Qlik Sense Data Connectivity

Discussion board where members can learn more about Qlik Sense Data Connectivity.

Announcements
QlikWorld 2020: Join us May 11 - 14, 2020 in Phoenix, AZ. Register early and save $400. Learn More
gandalfgray
Valued Contributor II

"Select data to load" in Data Load Editor fetches entire table

When we use the "Select data to load" in the Data Load Editor using a data connection via Amazon Redshift ODBC Sense sends a

SELECT * FROM <tablename> 

without any limit it seems.

One of our developers clicked on a large table and that query was running for 22 hours. 

Is there a way to make the "Select data to load" add a limit on what it fetches?

Labels (2)
1 Solution

Accepted Solutions
MVP & Luminary
MVP & Luminary

Re: "Select data to load" in Data Load Editor fetches entire table

I've found what QS supposedly does with the data preview: https://support.qlik.com/articles/000060939

A select count(*) is executed first. Perhaps Amazon Redshift doesn't do this very well and executes a full table scan.
Or QS tries to load only 100 records and QS does something wrong there or Amazon Redshift doesn't get it and tries to return everything.  Though I don't see really how that could happen in this age of cursors and pagination and whatnot. 


talk is cheap, supply exceeds demand

View solution in original post

5 Replies
MVP & Luminary
MVP & Luminary

Re: "Select data to load" in Data Load Editor fetches entire table

When developing you can use the Debug mode and in that mode choose to load a limited number of records.
Another way is to add a WHERE clause to the generated load statement:
SELECT * FROM <tablename> WHERE <... condition(s) to limit the data retrieved ...>

talk is cheap, supply exceeds demand
gandalfgray
Valued Contributor II

Re: "Select data to load" in Data Load Editor fetches entire table

Thanks Gysbert, but you are missing my point.

This happens when you pick a table from the "Select data to load" window (see below),

SelectDataWindow.PNG

that is, when you click on any of the tables in the table list.

 

I would expect Sense to be smart enough to just fetch a few 1000 rows or something, to be able to present a preview, but it seems it fetches the full table.

MVP & Luminary
MVP & Luminary

Re: "Select data to load" in Data Load Editor fetches entire table

In my experience the data preview does not load all the data from the source table.

talk is cheap, supply exceeds demand
Highlighted
gandalfgray
Valued Contributor II

Re: "Select data to load" in Data Load Editor fetches entire table

We usually don't have problems with SQLServer or PostgreSql data, but this is Amazon Redshift, and there it performs a full table load according to the logging
MVP & Luminary
MVP & Luminary

Re: "Select data to load" in Data Load Editor fetches entire table

I've found what QS supposedly does with the data preview: https://support.qlik.com/articles/000060939

A select count(*) is executed first. Perhaps Amazon Redshift doesn't do this very well and executes a full table scan.
Or QS tries to load only 100 records and QS does something wrong there or Amazon Redshift doesn't get it and tries to return everything.  Though I don't see really how that could happen in this age of cursors and pagination and whatnot. 


talk is cheap, supply exceeds demand

View solution in original post