Data loads can silently partially fail

[#1]

- We have noticed that data loads into QlikView Personal Edition can silently fail.

- That is, when we press control-R to reload data from a SQL Server database, we have seen cases where not all data is loaded, but no error message is displayed. We only notice this by carefully comparing counts of number of rows loaded versus counts in the corresponding database views.

- We have a .qvw file, which is storing data to .qvd files.

- Has anyone else seen this?

[#2]

No, we are not using Set ErrorMode.

[#3]

Have you made any progress on this issue.  I think we are experiencing something similar, and in our case it occurs when reloading data to a .qvw via an EDX call from a script within an SSIS package.  Over a period of 6 weeks it has occurred on three different .qvws.    Like the author of this thread, we receive no error indication either from the SQL Server/SSIS or from the Qlikview server.  We have only been able to uncover by performing row counts against the .qvw and comparing to row counts from corresponding table(s) in SQL DB.

The .qvw's that are failing to load are among our largest data tables.  Sometimes they load and sometimes they don't.  In our testing so far, if we re-run via the SSIS scripted EDX call the reload is sometimes successful and sometimes silently fails.  However so far if we reload directly through the Qlikview application we have not experienced this failure.

At this point, we are searching for any type of ideas as to the cause.

-Brookes

[#4]

We have not solved this issue yet, as its very interrmittent.

[marcus_sommer]

How earlier mentioned I suggest first to exclude defintily potential errors directly in values or expressions. Some of them could lead to ignore single records or to terminate the load/script without any message. Therefor I would add some routines which first count the rows in source and after loading counts again. Is there a different check about the source record-id which records are missing and write this result with a timestamp in a additionally table. Later you can check this table and search directly in this records which are missing to find any pattern.

Other reasons could be errors in RAM ... but this is more difficult. If you have several server in your environment you could move your app/task or execute it additionally parallel.

- Marcus

[saumyashah90]

this can happen when Set ErrorMode=0

[#7]

I will look into this.  We are also seeing this on a new server in a virtual environment and our systems team has made the decision to allocate additional memory to the SQL DB server.  Because of the way we process data into our SQL data warehouse tables and then into Qlikview documents, and as many as 7 SSIS packages at any one time are processing data into different SQL tables and then initiating a reload of Qlikview documents via scripted EDX calls, the systems admins believe that this may be overwhealming memory on the SQL DB server when our largest tables are processed.

[#8]

We don't use Set ErrorMode at all in the scrpts for the Qlikview documents which are silently failing.  Therefore according to my reading our scripts should be running in default ErrorMode = 1, however since we see no errors reported it sure seems like our scripts are running in ErrorMode = 0.  Is there another way this could have been set as a default if it does not appear in the Qlikview scripts?

[marcus_malinow]

If you're using MS SQL Server, have you tried adding the server hint (nolock) to your sql select statements?

E.g. SELECT * FROM TABLE (nolock)

If you're retrieving data at the same time that SQL Server is committing it you may find that some of your data is not being returned. Nolock may be able to help in this situation.

Marcus

[#10]

We use nolock. But this issue was being seen when no commits were being made.

[#11]

No solution for us yet, however we are being told that the issue may result from our use of SQL Server 2012 with Qlikview 10, which is not officially supported.  We are not sure the issue lies here, but our attempt to open an incident ticket with Qlikview was rejected as they will not troubleshoot an issue involving Qlikview 10 and SQL Server 2012.  We are evaluating our upgrade process, but we have yet to get the new EDX call working as we need it, and were trying to hold out for new server hardware before upgrading to Qlikview 11.  Now we will have to upgrade in place and then in a couple of months migrate to another server.