Skip to main content
Announcements
Qlik Connect 2025: 3 days of full immersion in data, analytics, and AI. May 13-15 | Orlando, FL: Learn More
cancel
Showing results for 
Search instead for 
Did you mean: 
vengadeshpalani
Creator
Creator

what is the best practice to load Multiple XML files

Hi all,

I want to load 18000 XML files on Daily basis each one has maximum 10000 rows what is the best practice to convert into single QVD.

in between, i want to perform Transformation Logic also.

3 Replies
prieper
Master II
Master II

Data: LOAD * INLINE [File];              // Dummy Table to enable Concatenation

FOR EACH sFile IN FILELIST ('MyFileList_w_*')

Data: CONCATENATE (Data) LOAD

..... // here come a brilliant script and transformation of data

FILENAME()          AS File          // just keep a reference

FROM

[$(sFile)]

(XmlSimple, Table is [.....]);

NEXT sFile

STORE Data INTO Data.qvd (QVD);

MK9885
Master II
Master II

All the fields in your 18000 XML files are same or they different?

marcus_sommer

I suggest rather if anyhow possible to avoid such approach and to load these data from a database because loading about 180 M records from "complex" text-files - xml is nearly the slowest possible file-format and a "normal" csv would be loaded a lot faster - will take some time ... especially if the data are distributed to so many files. If each load takes 0.5 seconds on overhead to establish the load-statement it would take 2.5 hours by 18 K files without loading, transforming and storing the data.

Further I think the implementation of an incrmental load approach would be quite useful. Here you will find various links to this topic: Advanced topics for creating a qlik datamodel.

- Marcus