Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Incremental load and full load

Dear community,

Due to the performance issue on massive data and high frequency of data load from database server (reload every 30 minutes),

we would like to use incremental load in every 30minutes, but to avoid losing of any data, we would like to have a full load once per week.

Is it possible to have incremental load and full load in the same document script?

PS: We are talking about the data - 1 million rows with 2Mb per row in 1 month growth,

and keep growing now.

Thanks,

Chanel

5 Replies
its_anandrjs

Yes possible check the Week days value and according to that per week activate that code on that day suppose every week Sat Evening or any day use. You have to use same script but in if conditions in first If part use every 30 minutes reload and then in second part use per week condition. By this you get the full load per week.

Hope this helps

Not applicable
Author

Thanks for the idea! but could you provide any sample or website link for this method?

so that i can try it out.

Thanks,

Chanel

Anonymous
Not applicable
Author

Hi,

Definitely, it is possible to have a incremental load and full load in the same document script. You just need to handle them with proper condition. For example, if weekday is sunday and time is in between 9am to 10am than run full load script else run incremental load script.

Please note, since this is running every 30min, you may face issue if the full load is not finishing within 30min.

its_anandrjs

You can search in the community about the incremental loads and use it with the variables.

Ex:-

Let vWeekDay = Weekday(Today( ) );

If $(vWeekDay) = 'Sat'; Then

Incremental Load Script every week logic

Else

Incremental Load Script every 30 minutes logic

End if

Not applicable
Author

Thanks for the important note!