2 Replies Latest reply: May 26, 2010 6:00 AM by tonymacp RSS

    Load data from lots of files

      Hi everybody,

      I am loading data from about 40 csv files, all of which have the same format and sit in the same folder. At the moment I am going through it 4 files at a time, specfying the file name and storing everything in a qvd as I go:

       


      MyTable:
      LOAD Field1,
      Field2,
      Field3
      FROM D:\MyFolder\File1.csv;
      CONCATENATE
      LOAD Field1,
      Field2,
      Field3
      FROM D:\MyFolder\File2.csv;
      etc...
      CONCATENATE
      LOAD Field1,
      Field2,
      Field3
      FROM D:\MyFolder\MyQVD.qvd (qvd);
      STORE MyTable INTO D:\MyFolder\MyQVD.qvd;




      After each run I edit the scipt to specify the file names of the next 4 csv files. Obviously, this is taking some time.

      I'm wondering if there's a way to do something along these lines:

       


      Set MyFolder = D:\MyFolder\
      For Each csvFile in MyFolder
      Set MyCSV = csvFile.Name
      MyTable:
      LOAD Field1,
      Field2,
      Field3
      FROM D:\MyFolder\MyCSV ;
      CONCATENATE
      LOAD Field1,
      Field2,
      Field3
      FROM D:\MyFolder\MyQVD.qvd (qvd);
      STORE MyTable INTO D:\MyFolder\MyQVD.qvd;
      Next


      Does that make any sense?

      Any suggestions gratefully received.

      Tony