Getting different number of records each time while loading manually

[Lucius_Artorius_Castus]

Hi, I am getting a different number of records each time while loading manually, like "293,632", "301,568" and correct is "310205", Now we have connected to dremio via Arrow flight odbc sql diver, so could it be driver issue?
Or qlik issue?

[Anil_Babu_Samineni]

did you checked if that is data issue? If data getting update obviously Qlik get the latest information from DB. If not, are you using any Rand() functions to get data generate? 

PS: till now I don't see that this is connector problem. 

[Lucius_Artorius_Castus]

@Anil_Babu_Samineni  I checked on dremio count is correct, but qlik is fetching different records.
So yesterday I scheduled one hour gap for the reload and wrote the script to store the the count for each time to know the changing frequency. and storing in qvf. and this is result.
FYI: These are the correct results 310205 for yesterday and today's count 310527 and in db it changes in night time and we are reloading after 5 or 6 hour buffer in qlik and in dremio I guess it is updating vds within 1hour after change in db.

[marcus_sommer]

The missing records might be locked through any other access against this and related tables. Therefore take a look on the data-base and driver settings, for example if you could use a read-only mode.  

[Lucius_Artorius_Castus]

@marcus_sommer No I don't think so that is happening, as you can see in screen shot it also showing correct count 310527 and and in next load count is decreasing. and if i load again it will come correct.

[marcus_sommer]

Essential is the understanding that Qlik doesn't execute any sql else it transferred only a string per driver to the data-base and gets reversed the resulting data-set. This means that any missing records are caused by the driver and/or the data-base.

The above mentioned lock of records shouldn't be too easily discarded because each kind of access may lead to it - of course any updates but also other read-accesses from other queries and/or backup/indexing-tasks - as well as any kind of timeouts or ending for higher priorities ...

Further thinkable are any kind of inconsistent access rights or data-handling but this would be probably one of the last possibilities which I would investigate.

If possible take a look in the log-files of the data-base + driver to find valuable hints. Beside this you may also try to track the relevant records, for example by adding recno() and rowno() in the load to identify the missing records and looking on them. Is any pattern noticeable - any invalide data or unexpected NULL's or it is always same kind / sub-set of data or anything else?

[Lucius_Artorius_Castus]

@marcus_sommer Ok I will try this, Thanks.