Qlik Community

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

Announcements
QlikWorld 2022, LIVE in Denver CO., May 16-19, 2022. REGISTER NOW TO RECEIVE EARLY BIRD PRICING
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Read big XML-Data

Hello together,

in a current project, I have to read big XML-Files (or a special XML-format, called "DDEX") with Qlikview.

A single file can be up to 5 GB big. When i start to load a file, it takes a pretty long before Qlikview (QV) starts to load the first dataset. One file takes approx. 1 hour with our testing-server to load, while almost 50 % of that time pass, without even loading a single dataset (Regarding to the "Skript-Status-Box")

Does anybody know, if QV checks the integrity of XML-Files first or does it take so long to analyse the structure of the file ? If QV verifys the XML-file, is it possible to skipt that step ? I can almost 100% gurantee that the files are valid.

Or does anybody have another idea to boost the performance ?

Thanks in advance and greetings

Florian

4 Replies
marcus_sommer
MVP & Luminary
MVP & Luminary

XML files will be always load quite slow compared with other formats like csv then the XML structures creates some overhead which needs to be parsed. And yes qlikview checked the structures within the XML which you could see if you creates a load-statement with the table-wizard and it will detect errors (surely not each typ but I have already seen error-messages that there was invalid characters/structures) and qlikview creates automatically keys between the tables.

But I don't know if this behaviour will be applied by normal (manually written) load-statements and if it's could be customized - maybe there are settings within the easter egg.

But by such large loadings you could improve load performance with incremental loads - here you will find various links to these topic: Advanced topics for creating a qlik datamodel.

- Marcus

petter
Partner
Partner

How does you load statement look like in the format specifier does it say:   xmlsimple or xmlsax? These two options are mutually exclusive and will use two very different xml parsers. The xmlsax option will use and require Microsoft's MSXML 3.0 or higher installed on the computer that runs the load script - this is almost always installed on any Windows anyway.

The xmlsax should be more suited to your heavy duty processing I would guess. But it all depends on what you will do with the XML-file and what kind of processing will be done. If you just want to get out some single "table" which is quite simple then maybe some "external" tool would be more efficient to use as a pre-processing step before you load it into QlikView.

petter
Partner
Partner

2015-10-01 #1.PNG

Not applicable
Author

Hey Petter,

i´m currently using XMLSimple, i´m gona try out xmlsax asap, thanks for your suggestion.

Unfortunately i have to load the whole XML-Doc.. It´s a loader-QVW which loads the whole XML-Structure and saves it in QVD-Files which are used in other QVWs afterwards. The workflow doens´t allow me to spare any data, because it´s the key to analyze these files if they are worthy to get imported in another system (long and expensiv process).