Qlik Community

Ask a Question

Connectivity & Data Prep

Discussion board where members can learn more about Qlik Sense Data Connectivity.

Announcements
LINKEDIN LIVE: Democratizing data to enhance customer-centricity. JULY 29TH REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
gandalfgray
Specialist II
Specialist II

"Select data to load" in Data Load Editor fetches entire table

When we use the "Select data to load" in the Data Load Editor using a data connection via Amazon Redshift ODBC Sense sends a

SELECT * FROM <tablename> 

without any limit it seems.

One of our developers clicked on a large table and that query was running for 22 hours. 

Is there a way to make the "Select data to load" add a limit on what it fetches?

Labels (2)
1 Solution

Accepted Solutions
Gysbert_Wassenaar

I've found what QS supposedly does with the data preview: https://support.qlik.com/articles/000060939

A select count(*) is executed first. Perhaps Amazon Redshift doesn't do this very well and executes a full table scan.
Or QS tries to load only 100 records and QS does something wrong there or Amazon Redshift doesn't get it and tries to return everything.  Though I don't see really how that could happen in this age of cursors and pagination and whatnot. 


talk is cheap, supply exceeds demand

View solution in original post

5 Replies
Gysbert_Wassenaar

When developing you can use the Debug mode and in that mode choose to load a limited number of records.
Another way is to add a WHERE clause to the generated load statement:
SELECT * FROM <tablename> WHERE <... condition(s) to limit the data retrieved ...>

talk is cheap, supply exceeds demand
gandalfgray
Specialist II
Specialist II
Author

Thanks Gysbert, but you are missing my point.

This happens when you pick a table from the "Select data to load" window (see below),

SelectDataWindow.PNG

that is, when you click on any of the tables in the table list.

 

I would expect Sense to be smart enough to just fetch a few 1000 rows or something, to be able to present a preview, but it seems it fetches the full table.

Gysbert_Wassenaar

In my experience the data preview does not load all the data from the source table.

talk is cheap, supply exceeds demand
gandalfgray
Specialist II
Specialist II
Author

We usually don't have problems with SQLServer or PostgreSql data, but this is Amazon Redshift, and there it performs a full table load according to the logging
Gysbert_Wassenaar

I've found what QS supposedly does with the data preview: https://support.qlik.com/articles/000060939

A select count(*) is executed first. Perhaps Amazon Redshift doesn't do this very well and executes a full table scan.
Or QS tries to load only 100 records and QS does something wrong there or Amazon Redshift doesn't get it and tries to return everything.  Though I don't see really how that could happen in this age of cursors and pagination and whatnot. 


talk is cheap, supply exceeds demand

View solution in original post