Read big XML-Data

[#1]

Hello together,

in a current project, I have to read big XML-Files (or a special XML-format, called "DDEX") with Qlikview.

A single file can be up to 5 GB big. When i start to load a file, it takes a pretty long before Qlikview (QV) starts to load the first dataset. One file takes approx. 1 hour with our testing-server to load, while almost 50 % of that time pass, without even loading a single dataset (Regarding to the "Skript-Status-Box")

Does anybody know, if QV checks the integrity of XML-Files first or does it take so long to analyse the structure of the file ? If QV verifys the XML-file, is it possible to skipt that step ? I can almost 100% gurantee that the files are valid.

Or does anybody have another idea to boost the performance ?

Thanks in advance and greetings

Florian

[marcus_sommer]

XML files will be always load quite slow compared with other formats like csv then the XML structures creates some overhead which needs to be parsed. And yes qlikview checked the structures within the XML which you could see if you creates a load-statement with the table-wizard and it will detect errors (surely not each typ but I have already seen error-messages that there was invalid characters/structures) and qlikview creates automatically keys between the tables.

But I don't know if this behaviour will be applied by normal (manually written) load-statements and if it's could be customized - maybe there are settings within the easter egg.

But by such large loadings you could improve load performance with incremental loads - here you will find various links to these topic: Advanced topics for creating a qlik datamodel.

- Marcus

[petter]

How does you load statement look like in the format specifier does it say:   xmlsimple or xmlsax? These two options are mutually exclusive and will use two very different xml parsers. The xmlsax option will use and require Microsoft's MSXML 3.0 or higher installed on the computer that runs the load script - this is almost always installed on any Windows anyway.

The xmlsax should be more suited to your heavy duty processing I would guess. But it all depends on what you will do with the XML-file and what kind of processing will be done. If you just want to get out some single "table" which is quite simple then maybe some "external" tool would be more efficient to use as a pre-processing step before you load it into QlikView.

[petter]

[#5]

Hey Petter,

i´m currently using XMLSimple, i´m gona try out xmlsax asap, thanks for your suggestion.

Unfortunately i have to load the whole XML-Doc.. It´s a loader-QVW which loads the whole XML-Structure and saves it in QVD-Files which are used in other QVWs afterwards. The workflow doens´t allow me to spare any data, because it´s the key to analyze these files if they are worthy to get imported in another system (long and expensiv process).