Loading New Data From QVSource

[#1]

Hi everyone,

I am trying to reload new data from some connectors (Facebook Fan Page and Twitter) using QVSource.

This is how it goes:

1) First reload, stores stuff into QVDs, good, I create a 'timestamp' field for each table, and set a variable to record the time of the update.

2) Try to reload new data into temp tables, where timestamp > variable. This does not work.

3) Assuming the above worked, I would concat my old QVDs to the new temp tables and store the new temp tables into the QVD.

I have tried setting the since_id parameter in the qvsource url to the variable, but to no success.

Where am I going wrong here?

[Anonymous]

can you share the code you are trying in 2

[MK_QSL]

What is your TimeStamp and Variable looks like?

[#4]

Let vLastUpdate = now();

Each TimeStamp is formatted as 'DD/MM/YYYY hh:mm:ss'

[MK_QSL]

If possible, provide your code .. .remove your actual field names with Field1, Field2 etc

[Anonymous]

now() is Today right.so I am not sure how we are downloading the future data.

Is your data like that.

Insteas of trying with variable you can try this

Timestamp > Today();

[#7]

Here is an example:

IF isnull(FileSize('filepath.qvd')) THEN

     Let vLastUpdate = now();

     FacebookFanPages

          Load *,

                  timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss') as timestamp

          From QVSourceURL;

          Store * from FaceBookFanPages into filepath.qvd (qvd);

ELSE

     FacebookFanPagesTemp:

          Load *,

                  timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss') as timestamp

          From QVSource URL where timestamp >= $(vLastUpdate);

         Concatenate

          Load * from filepath.qvd (qvd);

          Store * from FacebookFanPagesTemp into filepath.qvd (qvd);

          Drop table FacebookFanPagesTemp;

          Let vLastUpdate = now();

ENDIF

[its_anandrjs]

Not properly understood but i suggest use NOW() or what are the base format of this fields that you convert into timestamp.

[#9]

My understanding that when

Let vLastUpdate = now();

executes, vLastUpdate stays at the specific time. So when I reload I want only data that comes after vLastUpdate

[#10]

When reloading when QVDs are present, I get this message:

Field not found - <timestamp>. Am I not able to use a created field in a where clause?