Load only new values from a file

[#1]

Hello.

I have a file I need to load every month (e.g. a log) and this is the procedure I'm aiming to:

1. load old values stored in a qvd file
2. load the new files from the log
3. save the new table in a qvd file

Consider that in the log file I'm gonna import there could be rows already loaded in the qvd file, and I don't want duplicates.

Do you know any best practice to do what I'm looking for, please? I've tried with something like this, but with no luck:

LOAD id FROM <path>\file.qvd (qvd)
NewValues:
LOAD @1, @2 ... FROM <path>\log.txt WHERE NOT EXISTS(id)
CONCATENATE LOAD * FROM <path>\file.qvd (qvd)
STORE NewValues INTO <path>\file.qvd (qvd)

I always import all the values, or I get error stating that such table doesn't exist.

[#2 · Accepted solution]

Finally it seems I've found the way out for this problem. All your solutions were mainly correct, but I think that the main issue was related to the way of retrieveing data from the file:

LOAD if(...);
LOAD @1, @2... FROM path/file.log [...]

Now everything seems fixed with a code like this:

NewReport:
LOAD * FROM<path>/file.qvd (qvd);

LOAD
if (@1='04', @2, Peek('utente',-1)) As utente,
if (@1='05', @2) As tipologia,
if (@1='05', @3, Null()) As classe,
if (@1='05', @4, Null()) As numero_chiamato,
if (@1='05', Date(Date#(@5,'YYMMDD'),'DD/MM/YYYY'), Null()) As data,
if (@1='05', @6, Null()) As ora,
if (@1='05', @7, Null()) As durata,
if (@1='05', @8, Null()) As costo;

LOAD @1,
 @2,
 @3,
 @4,
 @5,
 @6,
 @7,
 @8
FROM <path>\test.dat (ansi, txt, delimiter is '\t', no labels)
WHERE (@1 = '04' OR @1 = '05');

LOAD *, utente&data&ora as ID /*** solving string ***/
RESIDENT NewReport;

STORE NewReport INTO <path>\file.qvd;

I still have to understand why it works even without the NOT EXISTS condition, but actually it does.

Thank you very much for the help!