7 Replies Latest reply: Sep 27, 2012 9:27 AM by John Lifter RSS

    loading Multiple Files in Directory

      Hi Guys,

      Need to load multiple files from a directory with extension .txt. How is this achieved in expressor?

        • Re: loading Multiple Files in Directory

          Yes this is something that can be done with Expressor. But it does require some coding in a Read Custom operator or in a script.

           

          Do all of the files have the same record structure?  That is, the same fields in the same order in each row. Do you want the processed data from each input file to be written to the same output, for example a common file, or to different targets?

            • Re: loading Multiple Files in Directory

              yes all the files have the same structure , and they will all be read into the same output file after transforms

               

              thanks,

              Paresh

                • Re: loading Multiple Files in Directory

                  Attach a sample file, or a few pseudo rows, and I can develop some sample code. Also, are you using the free download of Expressor or are you planning to purchase a license. There are, obviously, more options with a licensed version.

                    • Re: loading Multiple Files in Directory

                      Using a free download of expressor.

                       

                      columns in file

                      date,custno,amt

                      dd/mm/yyyy , int , decimal

                      there are around 15 files in csv format.

                       

                      Easily achieved in qlikview script using

                      for each file in filelist($(dir))

                      Load *

                      from File

                      Next;

                       

                      Using expressor to explore its metadata capabilities.

                        • Re: loading Multiple Files in Directory

                          Here is a screen shot showing the code you could use within the Read Custom operator to read all the rows from multiple .txt files in a directory, processing them into a single output file, for example, a .qvx file.

                           

                          read_custom_script.png

                           

                          In the initialize function, the code first extracts the directory location from an Expressor persistent variable and then uses the Windows type command line utility to concatentate the contents of the .txt files into a 'result set'. 

                           

                          In the read function, the code reads the first row from the 'result set'.  Then, what is termed an Expressor iterative return function, processes each row in the 'result set', parsing it into its individual fields, using each field to initialize an output attribute, and finally emitting the record. 

                           

                          Lines 13-22 repeat until all rows have been processed.  In lines 18-20, each field (all values are read from files as strings) is converted into its appropriate type.  When there are no more rows, the conditional statement on line 13 returns false and the else clause (lines 23-25) executes.  The return value expressor.ScriptSupport.FlowComplete indicates to the Expressor engine that processing has completed and the Read Custom operator may be destroyed.

                        • Re: loading Multiple Files in Directory

                          Hi John,

                           

                          Is data lineage metadata extraction possible from a expressor package deployment ( using the desktop version)?

                          e.g. i have very simple txt transformed into a qvx file. In the transformation artifact the is some datatype casting taking place. How do i extract that and present that to an end user?

                           

                          Regards,

                          Paresh

                            • Re: loading Multiple Files in Directory
                              Michael Tarallo

                              Hello Paresh  -

                               

                              Currently this is not available but as further integration continues between the two products, I suspect that it will be part of the QlikView Governance Dashboard as well as integrated into QlikView Expressor Desktop. So you are aware, the QlikView Governance Dashboard will be a free tool (available shortly) for QlikView customers to achieve visibility into their QlikView deployments. The QlikView Governance Dashboard does contain data lineage information in the current form for QlikView created and related data sources - it will show you what applications use which data sources as well as the derived expressions from those sources defined in the QlikView applications. So stay tuned.

                               

                              Regards,

                               

                              Mike T