Qlik Community

App Development

Discussion board where members can learn more about Qlik Sense App Development and Usage.

Announcements
Don't miss the upcoming Q&A with Qlik session on Qlik Application Automation on November 16th! REGISTER NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
iswarya
Partner
Partner

Incremental load in raw query

Hi,

I am using postgresql. I want to do a incremental load in qlik. But the issue here is, all the columns(ID,Name) are stored in a single column as Column1 in the raw query. 

After a full load, need to implement multiple transformation to get the fields(ID,Name) and then store it as final qvd.

How to achieve the incremental load for data like this?

4 Replies
stevedark
MVP
MVP

Hi @iswarya 

That's a strange format for database data to be in. If it is the case though, you need to use subfield to get all of the parts out.

LOAD
    subfield(Column1, ',', 1) as ID,
    subfield(Column1, ',', 2) as Name,
etc.

 

The problem that you will have though is that a WHERE statement in the SQL is likely to be inefficient anyway as the fields will not be properly indexed.

For general incremental load help, you may want to look here.

Steve

iswarya
Partner
Partner
Author

Hi @stevedark 

But I need to implement incremental load in the query.

stevedark
MVP
MVP

If the database needs to do a full table scan, due to not having an index, it will take as long to pull the last day or so of data as it will the whole table. In this case an incremental load will not be of any use.

Is it that the data in the database is cleared down so that you need to persist old data to QVD?

The article I posted a link to above has lots on information on incremental loads. There are many ways of achieving them and the best for you will depend on your requirements.

Steve

iswarya
Partner
Partner
Author

Yes, I need to persist old data to QVD.