1 Reply Latest reply: Feb 28, 2014 8:42 AM by John Lifter RSS

    READ FTP pipe delimitted file with UTF-8 encoding, LF record delimiter

      Hello,

        I am trying to read a pipe delimited file on another server.  I am using the readftp operator.  The file is encoded as UTF-8 with a LF as the record delimiter.  I want to create a comma delimitted .csv verison of this file on my local machine.

      How would I go about this?  I have tried various things. The readftp operator writes a .csv with only one LF at the end of the file.

        I'm using this code, but wonder how I do a line by line read.

       

      Thanks Traci

       

      flag = true

      curl = require "dscurl"

      results = {}

      c = nil

       

      function setCurl()

         results = {}

         c=curl.new()

         credentials = string.concatenate(

           _expCurrentOperatorParameters.username,":",_expCurrentOperatorParameters.password)

         c:setopt(curl.OPT_USERPWD,credentials)

         c:setopt(curl.OPT_WRITEDATA,results)

         c:setopt(curl.OPT_WRITEFUNCTION,

           function(results,buffer)

             table.insert(results,buffer)

             return #buffer

             end)

      end

       

      function initialize()

         setCurl()

      end

       

       

      function read()

         output = {}

         if flag then

           flag=false

           url = string.concatenate(

             "ftp://",_expCurrentOperatorParameters.server,"/",_expCurrentOperatorParameters.filename)

           c:setopt(curl.OPT_URL,url)

           c:perform()

          

           output.line=table.concat(results)

           log.notice(tostring(results))

           return output

         else

           return true

         end

      end

       

      function finalize()

         c:close()

      end

        • Re: READ FTP pipe delimitted file with UTF-8 encoding, LF record delimiter

          You likely have several issues here.  It would be a help to include a bit of the input and output files, but the following may help.

           

          As you read in each line and store it in the datascript table, you will be storing content such as:

          Word11|Word12|Work13

          Word21|Word22|Word23

           

          Therefore, you've done nothing to convert the pipe separated content into comma separated content.  What you might want to do before putting each line into the table is use string.replace to replace | with ,.  Something like: string.replace(buffer,"|",",") in the OPT_WRITEFUNCTION.

           

          Next, table.concat can receive a second parameter, which is the delimiter to place between each element of the table when concatenating into a string.  The default is no separator, so in your code table.concat will create one long string.  You might try entering "\r\n" as the separator; table.concat(results,"\r\n").

           

          Your output - output.line - should then include large string that when printed or written to file will include multiple lines of comma separated text.

           

          If you connect a Write File operator to the Read Custom operator and write the output to a file, you should then have a file with comma separated content.