Add data monthly without updating or deleting

[qv_bazz_10]

Hi guys,

i somwhow managed to load multiple xml (all with the same structure) into one qvW and it works as long as i always load all the original files.
Now i'd like to "save" all data loaded from those xml and store them into a qvd (or else?) to be able to add data from newer xml files.

This is my situation:

1. Every month i put hundreds of xml in a folder and the next month i create a new folder for the incoming files (so i have a folder per month)
2. Each xml contains a numerical value for every 15 minutes of any day of a certain month for the field "POD"

3. Many POD will have data every month (for different month) but not always and not all of them
4. I'd like to store all the xml in a folder in a qvd and monthly add the newer without updating or deleting anything (always append new data)
5. My script can read all the xml and splits data into two tables (one contains the values for every 15 minutes, the other one other informations)
6. I managed to store the info from the two tables in two qvd file (so i have 2 qvd per folder)
7. When i load the qvd from another script, the ones from the first folder are ok but the others are not (the POD with values from both folders seem to be multiplied and in different order).

What am i mistaking? Is there a better or easyer way to do what i'm triyng to do?

Hope i was clear enough for you to understand the problem.

Thanks in advance to everybody.

Andrea

[sudeepkm]

have you already tried using incremental load.

you just need to load everything from the files and then use a concatenate load from QVD.

also while storing the data in qvd you may also use a file name pattern to save the qvd along with date.

for the issue mentioned in #6 and #7 would you be able to share the app to verify?

[qv_bazz_10]

Hi Sudeep,

first of all, thanks for your reply. I was trying to use incremental load (or what i think incremental load is...) and my script was actually good.
My mistake was using autonumber to generate the keys to link the sections of the xmls and i managed to get it right using hash128.

Now i'm stuck with other two issues:

1. i can sum all the 96 loaded values with an expression after the scipt but can i do it with a variable in script to have it calculated during the execution of the script?
2. i'd like to group those 96 values by 4 (4 quarter of hour to have an hour) in script to have a 24 hours granularity of data, is it possible?

For example: the sum of quarters 1, 2, 3 and 4 gives me hour 1; the sum of quarters 5, 6, 7 and 8 gives me hour 2...

The names of my fields in script are:

E1 as Ea01,
E2 as Ea02,
...
E96 as Ea96

Thanks again for your advices.

Andrea

[sudeepkm · Accepted solution]

Do you get one XML file per day that has the field POD with 96 rows presenting values for every 15 min of that specific day?

You can run a for loop to concatenated load multiple such xml files and then another resident load can do an aggregation such as sum of values group by intervals such as by hour(i.e sum of 4 quarters) or per day.

[qv_bazz_10]

Sudeep,

i managed to do it, thanks for your useful advices!

Andrea