I would like to ask you how to write an 'Incremental Load" script which will load Hive table data filtered in the increments of seven days, such as:
where `date` = 18/1/2016, 25/1/2016, 1/2/2016.....17/9/2018, 24/9/2018, 1/10/2018, 8/10/2018.........onward
The script for the single date:
BN2: LOAD Distinct poi_region_name, state, efs_vc_host_name, Count (Distinct nbn_avc_id) as Unique_AVC_Count, tc4_11_avc_wo_ob_ds as TC4_download_speed, tc4_11_avc_wo_ob_us as TC4_upload_speed, tc1_11_avc_wo_ob_ds as TC1_download_speed, primary_access_tech, tc2_11_avc_wo_ob_ds as TC2_download_speed, tc1_11_avc_wo_ob_us as TC1_upload_speed, tc2_11_avc_wo_ob_us as TC2_upload_speed , Date(Date#("date", 'YYYYMMDD')) as Date where left (nbn_avc_id,3)='AVC' Group by poi_region_name,state,efs_vc_host_name,tc4_11_avc_wo_ob_ds, tc4_11_avc_wo_ob_us, tc1_11_avc_wo_ob_ds, primary_access_tech, tc2_11_avc_wo_ob_ds, tc1_11_avc_wo_ob_us, tc2_11_avc_wo_ob_us, "date";
I suggest not using the term incremental load as it has a specific usage within qlik.
for your usecase for dynamically creating the where clause. Why dont you create a variable in your script which will can be populated based on your logic and use in your where clause
Hi @dplr-rn , Could you please share more details about the type of incremental load you're looking to implement?
If your goal is to load data from the last week (i.e., the most recent 7 days based on the date), and then append it to the existing historical data, we can proceed as follows:
Create a variable: vDate = Date(Today() - 7) — this will return the date from 7 days ago.
Use this variable in your WHERE clause: WHERE Date = $(vDate)
Finally, concatenate the filtered data with your historical dataset.
