Qlik Community

QlikView Scripting

Discussion Board for collaboration on QlikView Scripting.

Not applicable

Loading New Data From QVSource

Hi everyone,

I am trying to reload new data from some connectors (Facebook Fan Page and Twitter) using QVSource.

This is how it goes:

1) First reload, stores stuff into QVDs, good, I create a 'timestamp' field for each table, and set a variable to record the time of the update.

2) Try to reload new data into temp tables, where timestamp > variable. This does not work.

3) Assuming the above worked, I would concat my old QVDs to the new temp tables and store the new temp tables into the QVD.

I have tried setting the since_id parameter in the qvsource url to the variable, but to no success.

Where am I going wrong here?

1 Solution

Accepted Solutions
jerem1234
Valued Contributor II

Re: Loading New Data From QVSource

When you say it does not work, do you mean it throws an error? I think it may boil down to this statement:


From QVSource URL where timestamp >= $(vLastUpdate);

I don't think it recognizes the field timestamp, since this is the field you are creating. Either try using the formula

where timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss')>= $(vLastUpdate)

Or maybe bring the data into the table again with a noconcatenate load, then use the condition :

where timestamp >= $(vLastUpdate);


Hope this helps!

11 Replies
jebamalai
Valued Contributor

Re: Loading New Data From QVSource

can you share the code you are trying in 2

Re: Loading New Data From QVSource

What is your TimeStamp and Variable looks like?

Not applicable

Re: Loading New Data From QVSource

Let vLastUpdate = now();

Each TimeStamp is formatted as 'DD/MM/YYYY hh:mm:ss'

Re: Loading New Data From QVSource

If possible, provide your code .. .remove your actual field names with Field1, Field2 etc

jebamalai
Valued Contributor

Re: Loading New Data From QVSource

now() is Today right.so I am not sure how we are downloading the future data.

Is your data like that.

Insteas of trying with variable you can try this

Timestamp > Today();

Not applicable

Re: Loading New Data From QVSource

Here is an example:

IF isnull(FileSize('filepath.qvd')) THEN

     Let vLastUpdate = now();

     FacebookFanPages

    

          Load *,

                  timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss') as timestamp

          From QVSourceURL;

          Store * from FaceBookFanPages into filepath.qvd (qvd);

ELSE

     FacebookFanPagesTemp:

          Load *,

                  timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss') as timestamp

          From QVSource URL where timestamp >= $(vLastUpdate);

         Concatenate

          Load * from filepath.qvd (qvd);

          Store * from FacebookFanPagesTemp into filepath.qvd (qvd);

          Drop table FacebookFanPagesTemp;

          Let vLastUpdate = now();

ENDIF

Re: Loading New Data From QVSource

Not properly understood but i suggest use NOW() or what are the base format of this fields that you convert into timestamp.

Not applicable

Re: Loading New Data From QVSource

My understanding that when

Let vLastUpdate = now();

executes, vLastUpdate stays at the specific time. So when I reload I want only data that comes after vLastUpdate

Not applicable

Re: Loading New Data From QVSource

When reloading when QVDs are present, I get this message:

Field not found - <timestamp>. Am I not able to use a created field in a where clause?

Community Browser