I am trying to do an incremental load using call center data. I do incremental loads frequently so this issue is baffling me. Every 30 minutes, a bulk insert of aggregate call center data is laoded into SQL. Each 30 minute batch is given the same timestamp. Ex. 3/27/13 20:00:000.
Alll I want to do is get the new records from the db and then concatenate what's in the QVD that I haven't just retrieved from the db. The logic works as expected, However, instead of returning all of the records from the QVD, it's only returning 1 record from each time interval instead of the whole batch.
For instance, if I had loaded 50 records at 19:30, I only get 1 record. The other 49 don't get loaded. Same with 19:00 - only record from that interval appears. And so on.
I am certain the batch inserts everything with the same timestamp. And I have also tested this by ignoring the time and just using the date. It's the same issue, all of the records for that date come in correctly from the database, but anything before that is still redcuded to record per interval.
I have even tried storing the field with a different name in the qvd so that there would be a clear distinction between field name being loaded from the db and the one that is stored in the QVD. The result is the same.
The issue seems to be related to the exists clause not working correctly.
Any thoughts?
