I think you will need an incremental approach with storing your results in a qvd and checking the next time if this file was already loaded and if yes the skip it. This should speed up your task a lot then the reading per filelist is quite fast and your xml-loadings will be the slower part.
I assume you need to recheck every file for appended lines?
I would probably create an external script that iterate over all files in that folderand writes the last line in a output file.
(e.g. using Power Shell or what ever you are familiar with)
Use EXECUTE statement in Qv to execute that script.
Then just read in the created output file.
Have you tried it with an incremental approach? Maybe something like this:
Load TEXT, FILE From YourQVD.qvd (qvd);
FOR Each vXML IN FileList ('Directory\*.xml')
if fieldindex('FILE', '$(vXML)') = 0 then
LOAD FirstSortedValue([@1:30],-RowNo()) AS TEXT, FileName() AS FILE
FROM [$(vXML)] (fix, codepage is 1252);
store TEST into YourQVD.qvd (qvd);
I did think about but it wouldn't work in this situation because the forms go back and forth between individuals. They make changes and it may not be corrupt today but the next person edits and saves causing the corruption.
After reading your post, I revisited this and will consider trying something related to Filetime(). Maybe if Today()-Filetime()>=8 instead of just looking for the filename. That way I can check only those changed in the previous week. I will post my results.