Hi everyone,
I've had a look and Googled the "issue" we currently have in our work environment, but haven't found an adequate answer to my question. Therefore, I've decided to kick off this new discussion point.
We have noticed our server maxi-ing out on CPU usage for an ETL only script these past few weeks that has been running for months without any issues. Since all the CPUs (server is an AWS r4.8xlarge) all max out at a 100% of (on all the 32) CPU usage, the machine freezes and all other tasks cannot load and the only solution is to completely reboot the machine, launch that one task and hope for the best.
However, I have noticed that for files that have a large amount of data (i.e. approx. 40 million rows in FACT table and 150 fields deep), when I try to dump the "ghost" data by "regenerating" the data content in the file (either via re-running the script with "exit script;" at the beginning or even just loading one INLINE table with only one row of data) and try and export the file via QMC, the declared file size if still extremely big, even though the initial "data trace" (for lack of a better term) has been overwritten.
One on test, the initial file which had previously loaded a 4Gb QVD (but has been dropped after a "STORE"), is still 800Mb without the data when I reloaded the script with an "exit script" at the start and then goes down to 180Mb when the file is reloaded with an INLINE table of one row of data. The original large QVD data is nowhere to be seen either by calling for the fields in the front-end or in the "Table Viewer". This file is a script only and has absolutely no front-end objects, stories, etc. that could explain this abnormal file size.
Does anyone know why this occurs? Has anyone witnessed the same and know how to resolve this? Will this change in future version (if Qlik Support team read this discussion point)?
Having had to manually check the file sizes of the script file via "export" function in QMC, I was able to find the "culprit". I have now copy/pasted all the tabs into a new virgin file, re-published it and the script seems to run fine.
Any help or thoughts would be greatly appreciated. Also, if this occurs on other files, is there a "nifty" tool for Sense to generate "QVS" files so that script and tabs can be moved easily from one Sense file to another?
Kind regards,
Cheenu
