On one of the .qvw - I'm using a simple query (select * from...) to load data and store it as a .qvd file for further usage.
Recently, I'm facing an issue with the retried data. Sometimes I get only some of the table's data and not all of it...
the document reloads every 2 hours (take few minutes) and some of the reload processes results with lack of data.
asking the ERP guy revealed that there is a process running on that table few times a day so we reduced it (in order to eliminate that issue) to once a day, when there is no .qvw reload process that run at that time.
Beside of any tasks which might run on the ERP could also read/writing-operations block some parts of the data (many databases have such features). To avoid the above described lost of data you could apply an incremental logic which read at first the qvd and add only the new or changed records to table which then again will be stored in a qvd. Here you will find several links to these topic: Advanced topics for creating a qlik datamodel.
I can't see how an Incremental Load will help me here. I have several ideas how to work around this issue.
That table in charge of the link between the ERP users and their Projects (some kind of a security table).
The whole table changes during the day. the difference between the resulted .qvd is (i guess) due to a lock at the DB (read\write operations on that table).
My question is if it's possible to sometimes load all of the data and some only part of it due to a different reason?
if the ERP guy says that the process that involved that table is now scheduled on different time now so there should be NO process involves with that table now.
Can it be from other reason? Or there must some other process (read\write) which still handling that table?
I think that there is any other process which locked (parts of) the table(s). Thinkable are also issues with the db-driver (instable?) or the network whereby these wouldn't be the first points which I would research. An indicator could be to check which records are missing - the first/last records which are rather related to driver, network or similar issues or records from the mid which indicate rather a locking of the records. If you research this very detailled you might be able to locate the real problem.
An incremental approach could be useful because if no new/changed records could be found no table/qvd/qvw would be changed and you would keep the last version. Maybe a simple check on the number of records would be enough to skip these update-part or to going further with them. If you add a timestamp (maybe in a variable) to your load-process you could display a gap between the reload-time and the last date/time of your records.
