Currency - store in QVD

[Anonymous]

Hi,

i'm trying to make a currency file, where I can store historical currencies.

The QVD file should grow every time I reload, if there is new data found.

I have found some code on this forum to get me started, but I'm stuck on how to store it in a QVD without getting an error.

For i = 0 to 18

    Let dd = Date(Today()-$(i), 'MM/DD/YY');

    Calendar:

    Load

         Date(Today()-$(i)) As ExchangeDate,

         Year(Today()-$(i)) As Year,

         Month(Today()-$(i)) As Month,

         Day(Today()-$(i)) As Day,

         Week(Today()-$(i)) As Week,

         WeekYear(Today()-$(i)) & '-' & Num(Week(Today()-$(i)),'00') As RollingWeek,

         Weekday(Today()-$(i)) As WeekDay,

         Date(MonthStart(Today()-$(i)),'YYYY-MM') As RollingMonth,

         Right(year(Today()),2) & '-Q' & ceil(Month(Today())/3)       as [YearQtr],

         'Q' & Ceil(Month(Today()-$(i))/3) As Quarter

    Autogenerate(1);    

    EuroRates:

    LOAD

     Date(Today()-$(i)) As ExchangeDate,

     Currency,

     Code,

     [EUR/1 Unit]            as ToEUR,

     [Units/1 EUR]            as FromEUR

    FROM

    [http://www.oanda.com/convert/fxdaily?date=$(dd)&date_fmt=us&exch=EUR&lang=en&sel_list=USD_HUF_UAH_RU...]

    (html, codepage is 1252, embedded labels, table is @1);

    Store EuroRates into QVD\EuroRates.QVD;

Next

If I let this code run, it just stops with the message "Execution of script failed. Reload old data?".

If I leave the Store EuroRates into QVD\EuroRates.QVD; line out of it, it loads currency data but only from the last date.

I'm expecting a different currency rate for each day, if it loads 18 days.

Could someone give me a hand to get this working?

Thank you!

[jonathandienst]

Evelien

I have a few suggestions for you to look at:

• The STORE statement is inside the loop. That means that the qvd will be overwritten on each pass, and only the last date will be stored. Try moving the STORE statement after the Next statement.
• If the STORE is causing an error, check that the folder exists. STORE does not create the folder.
• If that is not the problem, delete the qvd. Sometimes when QV terminates with an error, it leave a lock on the file. If you cannot delete the file, close all open QV files, go to task manager and terminate the qv process. Then try again.
• If you are still getting only one day's data, check that the OANDA url is returning the data you expect. If you step through in debugger, copy the generated URL and paste into your browser address bar.

Hope that helps

Jonathan

[Anonymous]

Johan,

thanks a lot for your suggestions.

I've put the Store statement after the Next statement. That works fine.

So now the loop works & gives me data for the number of days I put in the For i = 0 to ...

In the original statement I used 18 days.

If I reload with a number of days that is smaller than 18, some data dissappaers.

Is there a way to add data to a QVD file?

So that I can let it reload daily (automaticaly) and the new data is added?

Thanks!

Here, you find my adjusted code:

For i = 0 to 16

    Let dd = Date(Today()-$(i), 'MM/DD/YY');

    Calendar:

    Load

         Date(Today()-$(i)) As ExchangeDate,

         Year(Today()-$(i)) As Year,

         Month(Today()-$(i)) As Month,

         Day(Today()-$(i)) As Day,

         Week(Today()-$(i)) As Week,

         WeekYear(Today()-$(i)) & '-' & Num(Week(Today()-$(i)),'00') As RollingWeek,

         Weekday(Today()-$(i)) As WeekDay,

         Date(MonthStart(Today()-$(i)),'YYYY-MM') As RollingMonth,

         Right(year(Today()),2) & '-Q' & ceil(Month(Today())/3)       as [YearQtr],

         'Q' & Ceil(Month(Today()-$(i))/3) As Quarter

    Autogenerate(1);    

    EuroRates:

    LOAD

     Date(Today()-$(i)) As ExchangeDate,

     Currency,

     Code,

     [EUR/1 Unit]            as ToEUR,

     [Units/1 EUR]            as FromEUR

    FROM

     [http://www.oanda.com/convert/fxdaily?date=$(dd)&date_fmt=us&exch=EUR&lang=en&sel_list=USD_HUF_UAH_RU...]

    (html, codepage is 1252, embedded labels, table is @1);

Next

Store EuroRates into EuroRates.QVD;

[Anonymous]

Hi,

I found out that I have to use an incremental load to get the data added in the QVD.

After reading the reference manual, I'm not sure on how to build it up.

Anyone can help me on that?

[jonathandienst]

Evelien

Add the following code between the Next and the Store statements:

Concatenate (EuroRates)

LOAD * From EuroRates.qvd (qvd)

Where Not(Exists(ExchangeDate))

;

This loads the "old" qvd, but excludes dates that you loaded in your loop. So after this your table contains the old dates and the newly loaded ones.

Regards

Jonathan

[jonathandienst]

You could optimise this by loading data from the last Exchange date to today as follows:

T_LastDate:

LOAD Max(ExchangeDate) As MaxExchangeDate From EuroRates.qvd (qvd);

Let zMaxExchangeDate = Peek(MaxExchangeDate);

DROP Table T_LastDate;

For zi = zMaxExchangeDate + 1 To Today()

     Let dd = Date(zi, 'MM/DD/YY');

....

The rest of the script after Let dd is the same as before.

Regards

Jonathan

[Anonymous]

Johan,

I tried to paste your piece of code together with mine.

When I run it, I get this error:

Script line error:

For zi = zMaxExchangeDate + 1 To Today()

This is my current code:

T_LastDate:
LOAD 
Max(ExchangeDate) As MaxExchangeDate 
From EuroRates.qvd (qvd);
Let zMaxExchangeDate = Peek(MaxExchangeDate);
DROP Table T_LastDate;
For zi = zMaxExchangeDate + 1 to Today()
    Let dd = Date(zi, 'MM/DD/YY');
    EuroRates:
    LOAD
     Date(Today()-$(i))                                             As ExchangeDate,
     Currency, 
     Code, 
     [EUR/1 Unit]                                                    as ToEUR, 
     [Units/1 EUR]                                                    as FromEUR,
      Year(Today()-$(i)) As Year,
     Month(Today()-$(i)) As Month,
     Day(Today()-$(i)) As Day,
     Week(Today()-$(i)) As Week,
     WeekYear(Today()-$(i)) & '-' & Num(Week(Today()-$(i)),'00') As RollingWeek,
     Weekday(Today()-$(i)) As WeekDay,
     Date(MonthStart(Today()-$(i)),'YYYY-MM') As RollingMonth,
        Right(year(Today()),2) & '-Q' & ceil(Month(Today())/3)       as [YearQtr],
     'Q' & Ceil(Month(Today()-$(i))/3) As Quarter
    FROM
    [http://www.oanda.com/convert/fxdaily?date=$(dd)&date_fmt=us&exch=EUR&lang=en&sel_list=USD_HUF_UAH_RU...]
    (html, codepage is 1252, embedded labels, table is @1);
Next
Concatenate (EuroRates)
LOAD * From EuroRates.qvd (qvd)
Where Not(Exists(ExchangeDate));
Store EuroRates into EuroRates.QVD;

[jonathandienst]

Evelien

It looks like zMaxExchangeDate is not getting a value (ie is null).

Check that the Peek is working.Step through the script with the debugger, and check that zMaxExchangeDate is a number (about 41080).

This could happen if the qvd file is empty.

Regards

Jonathan

[#9]

Evelien (and Jonathan)

when you use incremental load, at first run (or in case your file was deleted) you will get an empty QVD file.

so i always start my sintax with the following:

IF isnull(QvdCreateTime('..\QVD\EuroRates.QVD')) THEN //************* New Load ***************

<your query> but with no date limit.

drop table EuroRates;

ELSE //************* Loads only new data ****************

<your query's last version - the one with the max date>

BTW,

after you finish and your query is up and running,

please post it here..... im interested in the final result.

thanks.

[Anonymous]

Jonathan

I will stop calling you 'Johan' from now, sorry for that.

The Eurorates QVD file is not empty & still it gives an null value for the peek(MaxExchangeDate).

Here you see a picture of the debugger.

Is this normal?