Qlik Community

Ask a Question

New to Qlik Sense

If you’re new to Qlik Sense, start with this Discussion Board and get up-to-speed quickly.

Announcements
QlikWorld Online 2021, May 10-12: Our Free, Virtual, Global Event REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
Pingcheater
Contributor
Contributor

Automatically load new data from CSV

Hi everyone,

hope this question is right here, sorry if not - I've just started working with Qlik.

So here is our current set-up:

We've got a system, that provides data over a period of time in a csv file. Currently, it's one line per 15 mins and each csv file only has the header and the current line. The csv files are stored in a FTP folder.

What we want is, Qlik to load those files into a QVD automatically and once loaded, to delete them in order for the QVD to be the main data source, not the csv files. The filenames include a date and timestamp (so can be predicted), but it would be even better, if Qlik would just load all files in the folder (e.g. in case the server is down for a moment, he would pick up all files, he missed) in order not to miss any data sets.

Is that understandable/ plausible?

Thank you in advance!

Labels (4)
2 Replies
jheasley
Luminary
Luminary

Yes - if I am understanding, you have something like the following:

5 files - SomeFile_Some_Time.CSV

all have the same fields. 

If so, then your load statement would be:

MainData:
Load
   Field1,
   Field2,
   Field3
From [filelocation/SomeFile_*.CSV]

then write a store command to store MainData into a QVD. When reloading from the QVD, you may want to do a load distinct to prevent duplicates. 

Pingcheater
Contributor
Contributor
Author

Thanks jheasley,

that's exactly, what I have.

Is there a way to delete the csv files after a successful load to avoid having unnecessary files on the store and speed up the load?