Incremental Load

[Anonymous]

Hi,

I would like to ask you how to write an 'Incremental Load" script which will load Hive table data  filtered  in the increments of seven days, such as:

where `date` = 18/1/2016, 25/1/2016, 1/2/2016.....17/9/2018, 24/9/2018, 1/10/2018, 8/10/2018.........onward

The script for the single date:

BN2:
LOAD Distinct
    poi_region_name,
    state,
    efs_vc_host_name,
    Count (Distinct nbn_avc_id) as Unique_AVC_Count,
    tc4_11_avc_wo_ob_ds as TC4_download_speed,
    tc4_11_avc_wo_ob_us as TC4_upload_speed,
    tc1_11_avc_wo_ob_ds as TC1_download_speed,
    primary_access_tech,
    tc2_11_avc_wo_ob_ds as TC2_download_speed,
    tc1_11_avc_wo_ob_us as TC1_upload_speed,
    tc2_11_avc_wo_ob_us as TC2_upload_speed ,
    Date(Date#("date", 'YYYYMMDD')) as Date
where left (nbn_avc_id,3)='AVC'
Group by poi_region_name,state,efs_vc_host_name,tc4_11_avc_wo_ob_ds,
    tc4_11_avc_wo_ob_us,
    tc1_11_avc_wo_ob_ds,
    primary_access_tech,
    tc2_11_avc_wo_ob_ds,
    tc1_11_avc_wo_ob_us,
    tc2_11_avc_wo_ob_us,
    "date";

SQL SELECT

    `poi_region_name`,
    state,
    `nbn_avc_id`,
    `tc4_11_avc_wo_ob_ds`,
    `tc4_11_avc_wo_ob_us`,
    `tc1_11_avc_wo_ob_ds`,
    `primary_access_tech`,

    `tc2_11_avc_wo_ob_ds`,
    `tc1_11_avc_wo_ob_us`,
    `tc2_11_avc_wo_ob_us`,

    `date`
FROM HIVE.prod.`baja`
where `date`='20180902';

Store BN2 into  [lib://QVD/BN2.csv]
(txt,delimiter is ',');

;

[dplr-rn]

I suggest not using the term incremental load as it has a specific usage within qlik.

for your usecase for dynamically creating the where clause. Why dont you create a variable in your script which will can be populated based on your logic and use in your where clause

e.g. where $(vTemp);