Loading New Data From QVSource

[#1]

Hi everyone,

I am trying to reload new data from some connectors (Facebook Fan Page and Twitter) using QVSource.

This is how it goes:

1) First reload, stores stuff into QVDs, good, I create a 'timestamp' field for each table, and set a variable to record the time of the update.

2) Try to reload new data into temp tables, where timestamp > variable. This does not work.

3) Assuming the above worked, I would concat my old QVDs to the new temp tables and store the new temp tables into the QVD.

I have tried setting the since_id parameter in the qvsource url to the variable, but to no success.

Where am I going wrong here?

[jerem1234 · Accepted solution]

When you say it does not work, do you mean it throws an error? I think it may boil down to this statement:

From QVSource URL where timestamp >= $(vLastUpdate);

I don't think it recognizes the field timestamp, since this is the field you are creating. Either try using the formula

where timestamp(Replace(subfield(created,'+',1),'T',' '),'MM/DD/YYYY hh:mm:ss')>= $(vLastUpdate)

Or maybe bring the data into the table again with a noconcatenate load, then use the condition :

where timestamp >= $(vLastUpdate);

Hope this helps!

[#3]

Thanks for the answer Jerem