Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi Guys,
I wanted to use daily incremental load. My source is SQL. can you help me how to insert daily records.
if i do a full load it would take max 4hrs to load. Also im getting issue when i try below steps;
given sample one for your understanding.
T1:
Load date from tbl1.qvd(qvd);
maxdate:
load max(date) as maxdate
resident T1;
let vlastupdateddate=peek(maxdate,0,'maxdate');
// tried let vlastupdateddate=peek('maxdate',0,'maxdate');
drop table T1;
Noconcatenate:
IncTable:
Load
a,b,date;
SQL select
a,b,date from hive.db.tbl1 where date > '$(vlastupdateddate)';
concatenate
Load a,b,date from tbl1.qvd(qvd);
store IncTable into ....filepath....tbl1.qvd(qvd);
after reloading , my data is not loaded with new records. tried giving date format in maxdate, in let variable too.
Appreciate your help.
Thanks in advance:)
You need to format the date in the correct date format for SQL source:
Let vlastupdateddate = Date(peek('maxdate'), 'YYYY/MM/DD');
Adjust the format to match your requirement. Check the log file to see the actual query sent to the DB.
Hi Jontydkpi,
i tried that one too.. still no records are fetched.
Any other script to perform incremental load using SQL select could also be fine.