How to identify within the script that app is already ran successfully?
Hi All,
I'm having an app that performs the ETL process (no client-side).
There is an initial process of creating QVD files that do not change later. I want to recognize that the app ran successfully so that in the following runs I will skip creating the files I mentioned above.
You may to consider to include a period-field and/or to include the period-information within the file-name. It may look like a lot of efforts to adjust a bigger part of your ETL but in the long-term it might be beneficial.
Beside this lookups are usually rather small tables - maybe you skip the whole incremental-stuff for them and running them each time without hitting any critical time-frame?
maybe you skip the whole incremental-stuff for them and running them each time without hitting any critical time-frame? - This is what I'm doing today - fully load these small tables every time.
I'm just trying to avoid this recreation again and again, But maybe the effort really isn’t worth it
In this case I wouldn't probably touch the logic like I load my standard-calendar each day which cost a second although once a year would enough to add/drop a year. It's some kind of noise floor which isn't essential in the overall processing and usually there are more relevant todo's on the list.
May be you include external config script to set up variables to indicate if you would like to reload the QVD in next run. 0 Values in case no need to run and 1 if QVDs are updated and need to be reloaded. I would try to update these variable dynamically if possible based on some process who decides that new QVDs to be reloaded.
