topic Re: Too slow fetching process in QlikView

Too slow fetching process

brunolelli87 — Wed, 25 Nov 2020 16:16:04 GMT

Hello guys,

I would like to better understand the loading process...

During the loading process of my script, I'm basically reading lots of Excel (.CSV) files, each one with something around 3000 rows and 30 columns. And Qlikview is doing it very quickly.

At the end of my script I'm loading another .CSV file with 400 rows and 10 columns, and it's taking too much time to load.

Please, take a look at the picture below:

While the system is loading the Cotação, I can't see the numbers changing, but when it comes to Dados, I can see the number improvement, like 240, 241, 242... 299, 300...

What is going on?
What can I do to speed it up?

Thanks
Bruno Lelli

Re: Too slow fetching process

Lisa_P — Mon, 25 May 2020 23:15:33 GMT

are you loading all of the data or doing some filtering ?

Re: Too slow fetching process

brunolelli87 — Mon, 25 May 2020 23:23:30 GMT

I'm loading all data, just removing the first row!

This is my script:

Dados:
LOAD

@1 as STCK, // Nome do papel
@1 as STCK2, // Nome do papel
@2 as EMPR, // Nome da empresa
@3 as SETO, // Setor da empresa
@4 as SUBS, // Subsetor da empresa
@5 as TIDA, // Tipo da ação negociada
@6 as NDAN // Número de ações negociadas

FROM
[https://www.lellidesp.com.br/Empresas/Empresas.csv]
(txt, utf8, no labels, delimiter is ';', msq, filters(
Remove(Row, Pos(Top, 1))
));

Re: Too slow fetching process

Lisa_P — Mon, 25 May 2020 23:55:33 GMT

It must be the internet connection ? Only takes me 2 seconds to load from Australia !

Re: Too slow fetching process

brunolelli87 — Tue, 26 May 2020 00:01:47 GMT

It's wierd, because I'm loading around 10 files with thousand of lines, and it's ok...

At the end, when I'm trying to load this small file, I'm getting this error.

Please, take a look at my project!
All data is stored online

Re: Too slow fetching process

Lisa_P — Tue, 26 May 2020 01:07:04 GMT

The time is not loading the fields, but the Replace Bundle load of storing the external image files inside the qvw.

See log file ..

2020-05-26 00:37:15 0523 Dados:
2020-05-26 00:37:15 0524 LOAD
2020-05-26 00:37:15 0525
2020-05-26 00:37:15 0526 @1 as STCK,
2020-05-26 00:37:15 0527 @1 as STCK2,
2020-05-26 00:37:15 0528 @2 as EMPR,
2020-05-26 00:37:15 0529 @3 as SETO,
2020-05-26 00:37:15 0530 @4 as SUBS,
2020-05-26 00:37:15 0531 @5 as TIDA,
2020-05-26 00:37:15 0532 @6 as NDAN
2020-05-26 00:37:15 0533
2020-05-26 00:37:15 0534 FROM
2020-05-26 00:37:15 0535 [https://www.lellidesp.com.br/Empresas/Empresas.csv]
2020-05-26 00:37:15 0536 (txt, utf8, no labels, delimiter is ';', msq, filters(
2020-05-26 00:37:15 0537 Remove(Row, Pos(Top, 1))
2020-05-26 00:37:15 0538 ))
2020-05-26 00:37:16 7 fields found: STCK, STCK2, EMPR, SETO, SUBS, TIDA, NDAN,
2020-05-26 00:37:16 404 lines fetched
2020-05-26 00:37:16 0540 Replace Bundle LOAD
2020-05-26 00:37:16 0541 @1 as STCK,
2020-05-26 00:37:16 0542 @8 as LDIM
2020-05-26 00:37:16 0543 FROM
2020-05-26 00:37:16 0544 [https://www.lellidesp.com.br/Empresas/Empresas.csv]
2020-05-26 00:37:16 0545 (txt, utf8, no labels, delimiter is ';', msq, filters(
2020-05-26 00:37:16 0546 Remove(Row, Pos(Top, 1))
2020-05-26 00:37:16 0547 ))
2020-05-26 00:45:48 2 fields found: STCK, LDIM,
2020-05-26 00:45:48 404 lines fetched
2020-05-26 00:45:49 Execution finished.

Re: Too slow fetching process

brunolelli87 — Tue, 26 May 2020 01:27:30 GMT

Uhnnn interesting!

Once all of those pictures are online... is there any way to speed it up?

Thanks