Is there a unique identifier which can help in identifying the existing records? If there is one, you can only load data which doesn't exist using Where not Exist statement. The logic you need is incremental load... Check these links to see how ti will work for you
I don't have direct access to the SQL data and there is no modify date.
The back end system is generating a csv file each day. The file name includes the date created.
So currently the task is setup to reload every csv file in the folder. There are currently 280 of them and eventually there will be over 1000.
There is a unique key (employee id).
Every new csv file contains new hires as well as has a record for each employee that was in the prior days csv file. Many of them are identical. Some might have updates. No real way to tell with the process we have currently except to compare every field and update if a change is found.
Is there a good way to do this when you can't access the data directly and are dealing with a daily csv file?
For doing this you need the following
- Any date field to identify that the record is modified (something like Modified/Updated Date field)
- Need a Primary key to identify each record
- Need to identify the data pattern
-Are we deleting any records, or just updating and inserting new records.
You can check for the sample script and examples in Qlikview help file.
Check for this topic
You will get very good explanation for all the scenarios.