Handling new fields added to data mart.

[#1]

Hello

We have 3 QVWs as follows: 1 reporting QVW, 1 modelling QVW and 1 data extraction QVW.

The data extraction QVW runs periodically and queries for (latest) data from our data mart. It performs some basic transformation on that data and then adds the data to an "incremental" QVD file. At the end of the month the incremental QVD becomes "historical" and a new incremental QVD is created. When the modelling QVW runs, it aggregates data from the last 26 monthly QVDs to produce our model.

My question is this: if someone adds a field to our data mart that I want to start including in our model, I obviously have to add it to the query of our data extraction QVW, but do I have to regenerate all of my (historical) monthly QVDs or is there a way to have my modelling QVW detect whether or not the new field exists in each monthly QVD during the aggregation process?

Thanks

Steve

[Anonymous]

This seems to simple to be what you're looking for....but would a Load * at some point in the process pick up the new field?

[#3]

Normally that would work just fine, but our modelling QVW loads the 26 QVDs inside a for-loop and from what I've seen, using Load * inside a loop can cause already loaded data to be reprocessed. I'll double check that though.

[Anonymous]

You could try loading from a for-each if that is possible

SET MODELED_Folder="[INSERT FOLDER PATH]"

;

FOR EACH file IN FILELIST('$(MODELED_Folder)\*.qvd')

LET file.ext = TextBetween('$(file)','\','.qvd',4)

;

LOAD

*

FROM $(file)(QVD)

;

NEXT file

[Gysbert_Wassenaar]

Qvd's are data files. They don't change if you don't recreate them. If you add a new field in your data model you need to regenerate the historical qvd files too if you want the new field in the historical qvd files.

[Anonymous]

Gysbert is right.

Just so I understand your issue:

- You have a process that incrementally builds the qvds each day from the data source.

- At the end of the month, the incremental qvds are stored (overwrite or appended?) to historical qvds.

- Each day, your data model builds using the historical qvds.

- If a new field is added to the data source, then it is included in the incremental qvd build.

- You want the historical qvd to identify if a new field exists within the incremental qvd.

Does this accurately reflect your question?

If so, it isn't clear to me what you want to occur when the historical identifies the new field(s). Will you elaborate on this aspect?