Incremental load in raw query

[iswarya]

Hi,

I am using postgresql. I want to do a incremental load in qlik. But the issue here is, all the columns(ID,Name) are stored in a single column as Column1 in the raw query. 

After a full load, need to implement multiple transformation to get the fields(ID,Name) and then store it as final qvd.

How to achieve the incremental load for data like this?

[stevedark]

Hi @iswarya 

That's a strange format for database data to be in. If it is the case though, you need to use subfield to get all of the parts out.

LOAD
    subfield(Column1, ',', 1) as ID,
    subfield(Column1, ',', 2) as Name,
etc.

The problem that you will have though is that a WHERE statement in the SQL is likely to be inefficient anyway as the fields will not be properly indexed.

For general incremental load help, you may want to look here.

Steve

[iswarya]

Hi @stevedark 

But I need to implement incremental load in the query.

[stevedark]

If the database needs to do a full table scan, due to not having an index, it will take as long to pull the last day or so of data as it will the whole table. In this case an incremental load will not be of any use.

Is it that the data in the database is cleared down so that you need to persist old data to QVD?

The article I posted a link to above has lots on information on incremental loads. There are many ways of achieving them and the best for you will depend on your requirements.

Steve

[iswarya]

Yes, I need to persist old data to QVD.