New Data

[qv_testing]

HI All,

Can we get New data without Incremental load.

like use last execution date..

I have million's of records.. How can we reload only new data?????

Like

Let vLastReload=  ReloadTime();

Load CategoryID,

        CategoryName,

        Price,

        Model

From Category;

Can we use variable in where Condition ??

Thanks in Advance...

[qv_testing]

I am getting this error.  and Application closing

Execution of Script failed. Reload old data??

[Kushal_Chawda]

Make sure that QVD path is correct

LET vToday = Date(Floor(ReloadTime()));

LET vQVDPath = 'D:\QVD\';

IF Alt(FileSize('$(vQVDPath)' & 'Data.qvd'), 0) > 0 THEN

  LET vLastReloadData = Date(Floor(ReloadTime()), 'MM/DD/YYYY');

ELSE

  LET vLastReloadData = Date(MakeDate(2014, 1), 'MM/DD/YYYY');

END IF

Table:

SELECT

     Category ID,

    Category,

    Product,

    DateCompleted,

    Price

FROM Category

WHERE DateCompleted>= TO_DATE('$(vLastReloadData)', 'MM/DD/YYYY');

IF Alt(FileSize('$(vQVDPath)' & 'Data.qvd'), 0) > 0 THEN

  Concatenate(Table)

  LOAD *

  FROM [$(vQVDPath)Data.qvd] (qvd)                        Note: 1st time i don't have this QVD

  WHERE DateCompleted< '$(vLastReloadData)';

END IF

STORE Data INTO [$(vQVDPath)Data.qvd];

DROP TABLE Table;

[jagan]

Hi,

Do you have this path?  D:\QVD\

Regards,

Jagan.

[qv_testing]

I got it..

QVD path it should be Double Quote's

[qv_testing]

Hi All,

I have Developed Incremental load, It's working fine..

I am Fetching from SQL Data....

But Full reload also taking same time..... Then what is the use of Incremental Load??????

Should I do anything change code??

Please give me suggestions.................

SET vQvdFile='Test.qvd';

SET vTableName='myTable';       

LET vQvdExists = if(FileSize('$(vQvdFile)') > 0, -1, 0);

IF $(vQvdExists) THEN          

          maxdateTab:

         LOAD max(DateField) as maxdate     

          FROM $(vQvdFile) (qvd);   

          LET vIncrementalExpression = 'WHERE DateField>=' & peek('maxdate');

          DROP table maxdateTab;

ELSE                                                      

          LET vIncrementalExpression = '';

END IF

$(vTableName):
LOAD
A,
DateField,
C SQL SELECT *
FROM DB.dbo."Table"
$(vIncrementalExpression);

IF $(vQvdExists) THEN

CONCATENATE ($(vTableName))

LOAD * FROM $(vQvdFile) (qvd);

End IF

STORE $(vTableName) INTO Test.qvd(qvd);

EXIT Script

If we have taking same time Incremental load and Full reload...

Then what is the use of Incremental Load....

Should i do any Optimization??

Thanks in Advance..

[qv_testing]

HI All,

I am fetching data from SQL. If i am reload full data, it's taking 12 mins.

I have implemented Incremental load for new records.

It's talking more than 50 mins.

Did i anything wrong??

Please let me know...

Thanks...

[settu_periasamy]

Hi,

Can you just check this part only? and check the duration..

$(vTableName):
LOAD
A,
DateField,
C SQL SELECT *
FROM DB.dbo."Table"
$(vIncrementalExpression);  //Manually give your condition

[qv_testing]

Hi Settu,

If i am give normal condition, it's reloading whole data.

Please help me out..

[settu_periasamy]

No, just wanted to check which part is taking much time.. Instead of using the variable (fetch data from SQL),

try to give the specific date..

is it possible to provide Log file?

Test:
LOAD
A,
DateField,
C;

SQL SELECT *

FROM DB.dbo."Table"
WHERE DateField>='1/1/2016';

[qv_testing]

This is working fine for Excel Data.

My SQL data date field format (like 23/04/2015 00:00:00)

is anything problem here??

Maxdate:

Load date(Max("Date"),'YYYY-MM-DD') as Maxdate

from Tablename;

LET vIncrementalExpression = 'WHERE DateField>=' & peek('maxdate');

Here i have converted Date YYYY-MM-DD,  but SQL data DD/MM/YYYY

Maybe compare issue.. if i am use that variable complete hang (lots of time).