More advanced control over data load required – where to start?

[davyqliks]

Hi All,

We have a relatively simple load script that pulls in text files (containing PO data).  We receive these files daily and the load script always starts from the very first file, importing them in 1 after the other.  These files are fairly large and after a year of business this reloaded is starting to take a while.  This is the first problem - we need to move to an incremental load so the newest files are added to the previously loaded data.  I believe we need to research incrementally loading to a QVD file Incremental reload and store into QVD file

However there's a second dimension to this query i want to understand how to deal with this when building the incremental loading script. 

So these files have recently had two new [additional] fields added to the end of the existing file format (all the original fields are still included).

We took the approach of duplicating the original load script and simply adding the two extra fields to it - which as i'm sure you can imagine has made a mess the tables.  QV has made links between every duplicate field - we have $syn tables galore!

So we need to learn how to do an incremental load but also how to bring in the new fields the existing table.

Would really appreciate a pointer as to where to start with this - which topics should we investigate?

Appreciate your help!

David

[marcus_sommer]

You could use a filelist-loop and force the concatenation of your files, like in this example:

Set vConcatenate = ;

For each vFile in FileList('.\*.txt')

Data:

$(vConcatenate)

LOAD * FROM [$(vFile)];

Set vConcatenate = Concatenate ;

Next vFile

- Marcus

[adamdavi3s]

I like that, neat!

You could (just for giggles) also do this which will load all text files without the loop

Data:

LOAD * INLINE [

ID

];

concatenate(Data)

LOAD *

FROM *.txt;

I did blog something about this before, the spec was to load only new files and keep a track of what had been loaded, probably way more complex than you need but it might be useful for some pointers

http://qlikanddirty.com/2016/08/26/loading-only-new-files-with-logfiles/