topic Re: Incremental load in raw query in App Development

Incremental load in raw query

iswarya — Wed, 02 Sep 2020 06:47:01 GMT

Hi,

I am using postgresql. I want to do a incremental load in qlik. But the issue here is, all the columns(ID,Name) are stored in a single column as Column1 in the raw query.

After a full load, need to implement multiple transformation to get the fields(ID,Name) and then store it as final qvd.

How to achieve the incremental load for data like this?

Re: Incremental load in raw query

stevedark — Wed, 02 Sep 2020 06:59:33 GMT

Hi @iswarya

That's a strange format for database data to be in. If it is the case though, you need to use subfield to get all of the parts out.

LOAD subfield(Column1, ',', 1) as ID, subfield(Column1, ',', 2) as Name, etc.

The problem that you will have though is that a WHERE statement in the SQL is likely to be inefficient anyway as the fields will not be properly indexed.

For general incremental load help, you may want to look here.

Steve

Re: Incremental load in raw query

iswarya — Thu, 03 Sep 2020 08:15:02 GMT

Hi @stevedark

But I need to implement incremental load in the query.

Re: Incremental load in raw query

stevedark — Thu, 03 Sep 2020 09:20:30 GMT

If the database needs to do a full table scan, due to not having an index, it will take as long to pull the last day or so of data as it will the whole table. In this case an incremental load will not be of any use.

Is it that the data in the database is cleared down so that you need to persist old data to QVD?

The article I posted a link to above has lots on information on incremental loads. There are many ways of achieving them and the best for you will depend on your requirements.

Steve

Re: Incremental load in raw query

iswarya — Thu, 03 Sep 2020 17:23:41 GMT

Yes, I need to persist old data to QVD.