I have a number of XML files that I loop over for data. The number of files used grows over time, and of course, looping over all of the files ever week is an inefficient way to load the data.
I'm trying to incorporate QVD files and an incremental load. I tried using both a Buffer Load and Buffer (Incremental) Load. In each case it creates a qvd file for each table created but as I loop over the files it removes the qvd files created previously.
I tried to explicitly store the info into a specified QVD file with a Concatenated Load but I had some trouble with it and didn't know if I was on the right track. I wonder if I should create a project that only appends new XML data to a QVD file. Perhaps that will simplify things.
So what is the best way to proceed with a growing list of XML files? Each new XML file is new data and doesn't modify existing data so I should be able to just append to the QVD file.
