Difference in Records b/w Qvd and database

[srg99999]

Hi guys,

I tried of loading the data from impala data base to QVD ,but i saw some difference in no of records b/w these two

records from impala database till 20/08/2019 ------ 8,872,799

records loaded into qvd till 20/08/2019----------------8,871,187

the difference is 1612 records.

And also am facing some issues while implementing the incremental loading.I think its due to some timestamp formatting issues 

[marcus_sommer]

If it's a live-db some records might be locked because any users/processes access them. Another point could be the driver which might not be able to transfer all records and doesn't run into an error else just skiped them. For the first mentioned possibility ask the db-admin if there is any way to bypass ist and for the second it's usually helpful to implement recno() and rowno() within a preceeding load of your sql to find those records - if it are always the same it indicates any issue with the data and if it's randomly it looked like locked records.

By your timestamps you might not only need a formatting else also a converting with timestamp#().

- Marcus

[srg99999]

Hi marcus,

Thank you so much for responding, my problem is i have 2620 different sensor  locations  with  24hours tracking.

so 2620 * 24 =62880 records i will get everyday as a new data. when i update that with qvd , i am getting 62880 records are adding  to existing qvd, but when i do a full load to test my incremental loading.,the full load is getting only some 27000 adding as a new data instead of 62880 .

[marcus_sommer]

Maybe there is an unsuitable where-clause by your loadings. Further how did you check the number of records? In the tableviewer or any UI object and is there any unique field (-combination) included ?

- Marcus