reload document stop after half of the records and script finished with no error

[QueenS]

Hi.

I'm facing a problem reloading a document.

the document script connect to database using oledb, and select all records from a table (~5 million records).

the script stops after 3 millions records and finished without an error.

I see memory usage is 99% for few minutes  before it stops. but still, no error is logged.

I tried adding to the script another load after this one and I got the error "connector error" upon reload. so i added another "oledb connect" to the script just before the second load and it succeeded the second table load - which means the script stops loading the first table in the middle because the opened connection to db was probably disconnected.

my question is why this is happening? is it because the memory was exhausted? and if so, why doesn't it dispay an error? if I didn't know how many records i have in the table, then i probably would not know  the model is not completely loaded.

in another server, with double size memory, it reloads fine.

i'll be happy to know how qliview behaves upon high memory usage. and how can I at least be notified in the logfile that the load was not complete.

[marksouzacosta]

Hi @QueenS,

How are you reloading your QlikView App? QMC or QlikView Desktop?
It is awkward to not see any Qlik Log. Have you checked the Windows Event Viewer? You may find something there.

Tip#1: On my ETL apps I generate an ETL Metadata file that counts how many records I have in the data source, and I how many I have in the QVD file. Later I can compare both.

Tip#2: Have you considered incremental load or data partition?

Regards,

Mark Costa

[QueenS]

I'm reloading through Qlikview Desktop. I also tried reloading in command line using Qv.exe and it is the same. There is a log for the reload but no errors or warnings are written in it.

Thanks for the idea to look in the Windows event viewer.

I do see a message in there, that the QVconnect was carshed. as I suspected:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: QvConnect64.EXE
P2: 0.0.0.0
P3: 00000000
P4: StackHash_3658
P5: 0.0.0.0
P6: 00000000
P7: c0000005
P8: PCH_D3_FROM_KERNELBASE+0x000000000003F19C
P9:
P10:

Thanks also for you tips. I do consider incremental log but I also want to figure out how to avoid this crash. 

I expect in my case that the app reload will take lots of time waiting for free memory, and not to crash the connection.

any idea if this can be handled some how?

[marksouzacosta]

In this scenario, I would try a mix of incremental load plus a batch processing.
The idea is to not consume much memory in the process, do not keep the connection opened for too long and if the app crashes for some reason, you can retake the process from where it left.

So, for instance, you can create a loop where you store the records into a QVD file every X number of records - let's say 100,000 records. The steps will be something like this:

1. Beginning of the loop: Connect to the Database
2. Collect how many records you have to process (what you have on your QVD file vs what is new/changed in your Database. That will change depending on your incremental load requirement)
3. If number of records is greater than zero, load the first 100,000 records
4. Disconnect from Database
5. End of the loop: Incrementally store the data in the QVD file
6. Repeat steps 1 to 5 until you have finished all the records you have to process

This is just the general idea of what you can do.

Regards,

Mark Costa