topic Re: Incremental Load in App Development

Incremental Load

Anonymous — Sun, 23 Sep 2018 13:38:25 GMT

Hi,

I would like to ask you how to write an 'Incremental Load" script which will load Hive table data filtered in the increments of seven days, such as:

where `date` = 18/1/2016, 25/1/2016, 1/2/2016.....17/9/2018, 24/9/2018, 1/10/2018, 8/10/2018.........onward

The script for the single date:

BN2:
LOAD Distinct
    poi_region_name,
    state,
    efs_vc_host_name,
    Count (Distinct nbn_avc_id) as Unique_AVC_Count,
    tc4_11_avc_wo_ob_ds as TC4_download_speed,
    tc4_11_avc_wo_ob_us as TC4_upload_speed,
    tc1_11_avc_wo_ob_ds as TC1_download_speed,
    primary_access_tech,
    tc2_11_avc_wo_ob_ds as TC2_download_speed,
    tc1_11_avc_wo_ob_us as TC1_upload_speed,
    tc2_11_avc_wo_ob_us as TC2_upload_speed ,
    Date(Date#("date", 'YYYYMMDD')) as Date
where left (nbn_avc_id,3)='AVC'
Group by poi_region_name,state,efs_vc_host_name,tc4_11_avc_wo_ob_ds,
    tc4_11_avc_wo_ob_us,
    tc1_11_avc_wo_ob_ds,
    primary_access_tech,
    tc2_11_avc_wo_ob_ds,
    tc1_11_avc_wo_ob_us,
    tc2_11_avc_wo_ob_us,
    "date";

SQL SELECT

    `poi_region_name`,
    state,
    `nbn_avc_id`,
    `tc4_11_avc_wo_ob_ds`,
    `tc4_11_avc_wo_ob_us`,
    `tc1_11_avc_wo_ob_ds`,
    `primary_access_tech`,

    `tc2_11_avc_wo_ob_ds`,
    `tc1_11_avc_wo_ob_us`,
    `tc2_11_avc_wo_ob_us`,

`date`
FROM HIVE.prod.`baja`
where `date`='20180902';

Store BN2 into [lib://QVD/BN2.csv]
(txt,delimiter is ',');

;

Re: Incremental Load

dplr-rn — Mon, 24 Sep 2018 17:18:32 GMT

I suggest not using the term incremental load as it has a specific usage within qlik.

for your usecase for dynamically creating the where clause. Why dont you create a variable in your script which will can be populated based on your logic and use in your where clause

e.g. where $(vTemp);

Re: Incremental Load

Amit_Prajapati — Thu, 09 Oct 2025 09:07:27 GMT

Hi @dplr-rn , Could you please share more details about the type of incremental load you're looking to implement?

If your goal is to load data from the last week (i.e., the most recent 7 days based on the date), and then append it to the existing historical data, we can proceed as follows:

Create a variable: vDate = Date(Today() - 7) — this will return the date from 7 days ago.
Use this variable in your WHERE clause: WHERE Date = $(vDate)
Finally, concatenate the filtered data with your historical dataset.