Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Incremental Load

Hi,

I would like to ask you how to write an 'Incremental Load" script which will load Hive table data  filtered  in the increments of seven days, such as:

where `date` = 18/1/2016, 25/1/2016, 1/2/2016.....17/9/2018, 24/9/2018, 1/10/2018, 8/10/2018.........onward

The script for the single date:

BN2:
LOAD Distinct
    poi_region_name,
    state,
    efs_vc_host_name,
    Count (Distinct nbn_avc_id) as Unique_AVC_Count,
    tc4_11_avc_wo_ob_ds as TC4_download_speed,
    tc4_11_avc_wo_ob_us as TC4_upload_speed,
    tc1_11_avc_wo_ob_ds as TC1_download_speed,
    primary_access_tech,
    tc2_11_avc_wo_ob_ds as TC2_download_speed,
    tc1_11_avc_wo_ob_us as TC1_upload_speed,
    tc2_11_avc_wo_ob_us as TC2_upload_speed ,
    Date(Date#("date", 'YYYYMMDD')) as Date
where left (nbn_avc_id,3)='AVC'
Group by poi_region_name,state,efs_vc_host_name,tc4_11_avc_wo_ob_ds,
    tc4_11_avc_wo_ob_us,
    tc1_11_avc_wo_ob_ds,
    primary_access_tech,
    tc2_11_avc_wo_ob_ds,
    tc1_11_avc_wo_ob_us,
    tc2_11_avc_wo_ob_us,
    "date";

SQL SELECT

    `poi_region_name`,
    state,
    `nbn_avc_id`,
    `tc4_11_avc_wo_ob_ds`,
    `tc4_11_avc_wo_ob_us`,
    `tc1_11_avc_wo_ob_ds`,
    `primary_access_tech`,

    `tc2_11_avc_wo_ob_ds`,
    `tc1_11_avc_wo_ob_us`,
    `tc2_11_avc_wo_ob_us`,

    `date`
FROM HIVE.prod.`baja`
where `date`='20180902';

Store BN2 into  [lib://QVD/BN2.csv]
(txt,delimiter is ',');

;

1 Reply
Channa
Specialist III
Specialist III

first you do full load

 

after you load with condition

WHERE RELEASE_DATE_H >= add_days(CURRENT_DATE ,-7);

Concatenate

with QVD file u did full load

you should need primary KEY

FROM [lib://QVD/FACT_DECLARATION.qvd]
(qvd)
WHERE NOT Exists(DECLARATION_ID);

try 

 

Channa