topic Re: Time Taken To load 20gb data into qlikview in QlikView

Time Taken To load 20gb data into qlikview

Anonymous — Wed, 25 Nov 2020 16:16:04 GMT

Hi,

How much time it will take to load 20gb data from database to Qlikview,

Can someone help me to figure out this.

Re: Time Taken To load 20gb data into qlikview

Gysbert_Wassenaar — Fri, 18 Mar 2016 09:04:39 GMT

That depends on lots of things. It's impossible to tell. If you have a machine with 4GB of ram then it will not load all that data ever. If you have a machine with an extremely slow harddisk it may take a very long time. If the 20GB of data are in separate xml files of 1KB average size it will take very long. If the 20GB is stored in one field in one qvd file on a super fast drive in a super fast machine with lots of fast ram then it can be read very fast. It all depends.

Re: Time Taken To load 20gb data into qlikview

Chanty4u — Fri, 18 Mar 2016 09:06:24 GMT

chk dis

RAM and CPU Utilization%

CPU Utilization and Memory Utilization?

Re: Time Taken To load 20gb data into qlikview

Anonymous — Fri, 18 Mar 2016 09:54:34 GMT

Thanks Gysbert, But if I have 126 GB RAM , and I am fetching a Table from DB which is of size 20gb,

how much approx. time it may take.

Re: Time Taken To load 20gb data into qlikview

oknotsen — Fri, 18 Mar 2016 10:32:01 GMT

Strongly depends on what you do with the data, the columns, the distinct values, network connection, etc.

Data is calculated in rows, columns and distinct values, not in gb.

Re: Time Taken To load 20gb data into qlikview

Gysbert_Wassenaar — Fri, 18 Mar 2016 10:41:23 GMT

Impossible to tell. Is the table read from a file or from clay tablets? Is it read through a database connection or are the bits delivered using the RFC 2459 protocol * ? There are a thousand things that can influence the time.

* (ok, that's extremely unlikely, but still... as an example)

Try to make an estimate by loading the first 100.000 records:

OLEDB CONNECT TO .... etc;

LET vStartTime = Now(1);

FIRST 100000

SQL SELECT * FROM MyTable;

LET vTotalTime = Interval(Now(1) - $(vStartTime),'hh:mm:ss.fff');

TRACE $(vTotalTime);

If you know how many records your 20GB contains you can calculated how long loading all the data will approximately take.

Re: Time Taken To load 20gb data into qlikview

Kushal_Chawda — Fri, 18 Mar 2016 11:12:20 GMT

We can't say exactly how much time it will take to load the data. It depends on how fast is your database and how good is the network bandwidth between your QV server and database server. So try to load the data from machines with the different configurations and check which is the good machine to go for. To check the time to load the data you can try below.

ODBC .....

let vStartTime =now();

Table:

SQL SELECT *

FROM Source;

let vEndTime =now();

let vLoadingTime = interval('$(vEndTime )' - '$(vStartTime )','hh:mm:ss');

Check the value of variable vLoadingTime in input box

Re: Time Taken To load 20gb data into qlikview

Anonymous — Fri, 18 Mar 2016 12:57:38 GMT

Thank you so much Gysbert.

Re: Time Taken To load 20gb data into qlikview

Anonymous — Fri, 18 Mar 2016 12:57:49 GMT

Thank you so much.

Re: Time Taken To load 20gb data into qlikview

oknotsen — Fri, 18 Mar 2016 23:28:43 GMT

Or simply check the time on the progress screen at the end.

Re: Time Taken To load 20gb data into qlikview

Kushal_Chawda — Sat, 19 Mar 2016 05:07:21 GMT

Please mark the relevant answer as correct and close the thread.