Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
I am using postgresql. I want to do a incremental load in qlik. But the issue here is, all the columns(ID,Name) are stored in a single column as Column1 in the raw query.
After a full load, need to implement multiple transformation to get the fields(ID,Name) and then store it as final qvd.
How to achieve the incremental load for data like this?
Hi @iswarya
That's a strange format for database data to be in. If it is the case though, you need to use subfield to get all of the parts out.
LOAD
subfield(Column1, ',', 1) as ID,
subfield(Column1, ',', 2) as Name,
etc.
The problem that you will have though is that a WHERE statement in the SQL is likely to be inefficient anyway as the fields will not be properly indexed.
For general incremental load help, you may want to look here.
Steve
Hi @stevedark
But I need to implement incremental load in the query.
If the database needs to do a full table scan, due to not having an index, it will take as long to pull the last day or so of data as it will the whole table. In this case an incremental load will not be of any use.
Is it that the data in the database is cleared down so that you need to persist old data to QVD?
The article I posted a link to above has lots on information on incremental loads. There are many ways of achieving them and the best for you will depend on your requirements.
Steve
Yes, I need to persist old data to QVD.