Reading the latest data:

odaher123 · ‎2018-11-05

Hi guys,

Every day an application saves new data in new files (csv).

How to read only the data from the last files deposited since the last reading?

I managed to create a table named Listing_of_files.

Thanks for your help

/////////////////////////////////////////////////////////////////////////

set file_path_csv = 'F:\Statistiques\csv';

sub DoDir (Root)

For Each Ext in 'qvd', 'csv'

For Each File in filelist (Root&'\*.' &Ext)

Listing_of_files:

LOAD

'$(File)' as Name, FileSize( '$(File)' ) as

Size, FileTime( '$(File)' ) as FileTime

autogenerate 1;

Next File

Next Ext

For Each Dir in dirlist (Root&'\*' )

Call DoDir (Dir)

Next Dir

End Sub

Call DoDir ('$(file_path_csv)');

///////////////////////////////////////////////////

how to do next?

dplr-rn · ‎2018-11-14

Step 1 - Set up the csv file load (i.e. load only the new files)

Basically for the first load (based on a flag) load the file names and data into a qvds

Subsequent loads

- get list of files names

- load old list of file names from qvd

- create another table with latest filenames only

- load data from qvd

- loop through new file names and load which gets autoconcatenated

- drop unnecessary tables (havent done this in the app to prove it works)

- save qvds

see attached script

Step 2 - implement incremental load (update and/or) delete on the latest loads. check below