Skip to main content
Announcements
Customer Spotlight: Discover what’s possible with embedded analytics Oct. 16 at 10:00 AM ET: REGISTER NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Partial Reload accumulating data but based on the same source file changing

Hello,

i`m facing with the problem how to code accumulative partial reloads where source of new data is
the same file named  'src.csv' and is ovverwiritten by other process with new data every hour. QV application
should start partial reload every hour and appending new content of 'src.csv' (adding to data info what time
they were generated) but still keeping old data but not older than 14 days.


  I`ve coded so far sth like below:

SET HistoricDays = 14;

LET ftime = FileTime('src.csv');

LET minDate = Date#((Date(today())-$(HistoricDays)),'YYYY-MM-DD'); // lowest historical date to keep in datamodel at parial reload

//--------------------------------------------------------------------------------------

// load incremental data from changing src.csv file

Departures_tmp:

add LOAD

'$(ftime)' as D_CREATION_DATE_TIME,

  Date#(left('$(ftime)',10),'YYYY-MM-DD') as D_CREATION_DATE,

  data1,

  data2,

  ...

FROM

src.csv

(txt, codepage is 1250, no labels, delimiter is ';', msq);


// remove data older than 14 days (minDate) from the table at partial reload.

Departures_tmp2:

replace LOAD

D_CREATION_DATE_TIME,

D_CREATION_DATE,

data1,

data2,

...

RESIDENT Departures_tmp where Date#(D_CREATION_DATE) > $(minDate);

DROP TABLE Departures_tmp;

//--------------------------------------------------------------------------------------

When I trigger 'partial reload':

add LOAD works fine and appends new data into existing ones.

but 'replace LOAD' seems does not work , it looks like it loads only 'fresh' data from src.csv
(but oryginaly it bases on RESIDENT Departures_tmp table). I`ve tried replace 'replace LOAD' with 'add LOAD'
but it does not work eather.

Does any one have a good Idea to overcome this ?

Cheers.


					
				
			
			
				
			
			
			
			
			
			
			
		
0 Replies