topic Data loss when loading from local file in QlikView

Data loss when loading from local file

Tue, 14 Jun 2011 10:34:36 GMT

Hi,

I am trying to read in a flat file with 210K rows and 45 columns. Somehow I only get in about 117K rows (and all columns). My file is Unicode format (UTF-8) and the load script is as follows:

Table:

LOAD *

FROM

FILE.TXT(txt, utf8, embedded labels, delimiter is '|', msq, header is 7 lines, filters(

Remove(Row, Pos(Top, 2)),

Remove(Col, Pos(Top, 1))

));

I do a transformation of the data removing couple of extra rows and columns and selecting the header is on 7th row. However, I have also tried the load using an edited file, where this transformation is not necessary, the same end result.

Is there possibly a limitation to maximum rows in the QlikView load script, that I haven't figured out?

Data loss when loading from local file

Tue, 14 Jun 2011 10:55:21 GMT

EDIT: Solved, I tried this over and over again and changed the Quoting setting from MSQ -> Standard. Somehow that caused the loss of data...

Data loss when loading from local file

stephencredmond — Tue, 14 Jun 2011 10:56:24 GMT

Hi,

Not a limitation, it is just a problem with the encoding of the file and the quoting used. Try changing from msq to standard.

Regards,

Stephen

Data loss when loading from local file

stephencredmond — Tue, 14 Jun 2011 10:58:10 GMT

Glad you found it.

BTW - this is not a "loss" of data. It just gets to a certain point where the msq quoting indicates and end of file. So you end up only getting the first XXX lines. It just doesn't get to the end of the file.

Regards,

Stephen

Data loss when loading from local file

kji — Tue, 14 Jun 2011 11:01:17 GMT

The cause is generally an unbalanced quote charater on a line making the rest of the file being treated as one record, MSQ supports multiline data by this method.