topic Re: Loading New Data From QVSource in QlikView

Loading New Data From QVSource

Wed, 18 Jun 2014 13:20:24 GMT

Hi everyone,

I am trying to reload new data from some connectors (Facebook Fan Page and Twitter) using QVSource.

This is how it goes:

1) First reload, stores stuff into QVDs, good, I create a 'timestamp' field for each table, and set a variable to record the time of the update.

2) Try to reload new data into temp tables, where timestamp > variable. This does not work.

3) Assuming the above worked, I would concat my old QVDs to the new temp tables and store the new temp tables into the QVD.

I have tried setting the since_id parameter in the qvsource url to the variable, but to no success.

Where am I going wrong here?

Re: Loading New Data From QVSource

Anonymous — Wed, 18 Jun 2014 13:23:06 GMT

can you share the code you are trying in 2

Re: Loading New Data From QVSource

MK_QSL — Wed, 18 Jun 2014 13:24:40 GMT

What is your TimeStamp and Variable looks like?

Re: Loading New Data From QVSource

Wed, 18 Jun 2014 13:28:06 GMT

Let vLastUpdate = now();

Each TimeStamp is formatted as 'DD/MM/YYYY hh:mm:ss'

Re: Loading New Data From QVSource

MK_QSL — Wed, 18 Jun 2014 13:30:25 GMT

If possible, provide your code .. .remove your actual field names with Field1, Field2 etc

Re: Loading New Data From QVSource

Anonymous — Wed, 18 Jun 2014 13:33:00 GMT

now() is Today right.so I am not sure how we are downloading the future data.

Is your data like that.

Insteas of trying with variable you can try this

Timestamp > Today();

Re: Loading New Data From QVSource

Wed, 18 Jun 2014 13:38:14 GMT

Here is an example:

IF isnull(FileSize('filepath.qvd')) THEN

Let vLastUpdate = now();

FacebookFanPages

Load *,

timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss') as timestamp

From QVSourceURL;

Store * from FaceBookFanPages into filepath.qvd (qvd);

ELSE

FacebookFanPagesTemp:

Load *,

timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss') as timestamp

From QVSource URL where timestamp >= $(vLastUpdate);

Concatenate

Load * from filepath.qvd (qvd);

Store * from FacebookFanPagesTemp into filepath.qvd (qvd);

Drop table FacebookFanPagesTemp;

Let vLastUpdate = now();

ENDIF

Re: Loading New Data From QVSource

its_anandrjs — Wed, 18 Jun 2014 13:38:55 GMT

Not properly understood but i suggest use NOW() or what are the base format of this fields that you convert into timestamp.

Re: Loading New Data From QVSource

Wed, 18 Jun 2014 13:39:43 GMT

My understanding that when

Let vLastUpdate = now();

executes, vLastUpdate stays at the specific time. So when I reload I want only data that comes after vLastUpdate

Re: Loading New Data From QVSource

Wed, 18 Jun 2014 13:50:55 GMT

When reloading when QVDs are present, I get this message:

Field not found - <timestamp>. Am I not able to use a created field in a where clause?

Re: Loading New Data From QVSource

jerem1234 — Wed, 18 Jun 2014 13:51:32 GMT

When you say it does not work, do you mean it throws an error? I think it may boil down to this statement:

From QVSource URL where timestamp >= $(vLastUpdate);

I don't think it recognizes the field timestamp, since this is the field you are creating. Either try using the formula

where timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss')>= $(vLastUpdate)

Or maybe bring the data into the table again with a noconcatenate load, then use the condition :

where timestamp >= $(vLastUpdate);

Hope this helps!

Re: Loading New Data From QVSource

Wed, 18 Jun 2014 14:09:56 GMT

Thanks for the answer Jerem