Count of symbols in one field?

[Vasiliy_Beshentsev]

Hello, everyone today!

I have a problem with loadind log file.

So first of all i have a big log file, about 4 mb.

It's got more than 16000 rows in.

I'm loadind data like this, 16000+ rows in one field.

I'm already write handler for this 1 row.

So here is my question: WHY QLIK SENSE CUTTING MY FILE?!

It's not full load of my file! I'm write that 1 record is 16000+ lines but after loadind it's NOT!

How i get the point? Take a closer look.

My last record after working handle is

Let's take a look where is this record in file in NOTEPAD:

Why it's not full load of file?

[Vasiliy_Beshentsev]

I found that in my 1 row it load only 680992 symbols.

[Vasiliy_Beshentsev]

All file got 2 725 462 simbols (looking in MS Word)

[Vasiliy_Beshentsev]

I found that if copy/paste all data from log file into txt file it become less weight (2 mb) and now count of symbols in 1 row is became 1 000 001

[rwunderlich]

Could you explain what you mean by "symbols"? Do you mean distinct values? can you post your complete script?

If you are loading all rows into a single field and then processing that field, you may be running into some length restrictions depending of what statements you are using to process it.

-Rob

[Vasiliy_Beshentsev]

Hello, Rob, thanks for join discuss!

When i say "symbols" i mean count of chars in 1 row (in this example).

Or when you use Len(Field) - you can get this count.

So, when i load data from file i use this scripts:

1) It's the last, it paste into 1 row 1 000 001 chars.

2) It's the first load script, that i use, it give me 680 992 chars in 1 row.

(Do not be confused by filenames, i just copy all from log into txt)

[rwunderlich]

Try adding "no eof" to your format specification and see if that makes a difference.  I've never tried to read that many ( 2 725 462) bytes into a single field, so you may have other problems as well.  Can you process this as individual rows? If you are trying to tie multiple lines together, you can do that by assigning a logical record number to each and use concat() to group them back together,

-Rob

[Vasiliy_Beshentsev]

Thanks, no eof does nothing.

How to assign this logical number to each row?

This what i get when load as individual rows:

I need to concat by this groups, how to do that?

[Vasiliy_Beshentsev]

Made this:

So what next? Please, help

[marcus_sommer]

I think Rob is right and you shouldn't load everything within a single field-value else loading them row by row, maybe with an included recno(), rowno() and maybe some further counter per peek() or previous() in a following ordered resident load or if the datastructure is consistent (or could be mode) with something like:

mod(rowno(), 12) as FieldID,

ceil(rowno() / 12) as RecordID

which I used for a logfile which had a stream-format and each 12 rows belonged to one record. If this isn't helpful in any way please elaborate what is the intention to keep the whole content within a single field-value.

- Marcus