Huge Data Load

[anushree1]

Hi,

I am trying to load a table of 206 Columns having 16 million records from SAS through Qlikview Desktop.

Its seen that the first 3 Million records get loaded within the 3 Mins but next 1 million does not load even after 2 hrs and qlikview gets shut down , apart from the RAM and disk space could there be any issue causing this trouble

[Peter_Cammaert]

IMHO development machines (PCs) should have as much RAM as possible. Otherwise you'll have to make compromises or implement reductions as Avinash amply explained.

It's a different story for a server where you'll usually have enough RAM but you'll need to find a way to share it with other stakeholders.

[Edit] Ok, that depends. In my region, cost of RAM is way lower than developer time. YMMV.

[vardhancse]

Hi Anushree,

In real time not required to develop the dashboard with complete volume of data.

As Avinash suggested, best practice is to load some limited set of data and complete the presentation layer.

Post completion of build you can remove the filter in script and place the file in server and load complete data.

[anushree1]

I very much agree to it but then the requirements have not freezed yet.I keep getting changes and enhancements to be worked upon and moreover validation is also going on the other end so they need the complete data set with all the requirements as and when its worked upon.

[avinashelite]

As suggested earlier , either you increase your RAM or follow the approach which I specified . You could add all the UI components with limited load and then push the file to Server and reload ...I am not getting whats the issues with this approach

[vardhancse]

Hi Anushree,

Business/domain team will not be aware of the technical aspects they will just expect the output.

as a developer its our responsibility to make them understand the technical feasibility.

Moreover never its suggested to load whole set of data and do the build and then place in dashboard.

In that case we need to work in server, as it will have huge RAM and now we can able to load huge volume of data