Storing final QVD taking more time......

[paulwalker]

Hello experts,

I'm facing one issue...

I have 15 tables, have concatenated all those tables into one table... and storing data into one QVD.

these all concatenate process taking 5 to 10 min,  but after that store command taking too long.

Please can you help on this guys. ..

Thanks in Advance!!!

[Peter_Cammaert]

What happens to RAM consumption during your script run? After concatenating those tables into one, do you free up the originals? A STORE command is pretty memory-hungry, so making available sufficient RAM will help.

Also keep in mind that RAM-based operations are orders of magnitude fasyter than writing to external storage (even SSDs are slow in that respect).

[paulwalker]

Thanks for quick reply...

I have 8GB RAM,  without store command it's taking hardly 7 min and taking performance memory 1.3GB.

While creating QVD taking performance memory 7.7.8GB.

Is there any problem with NULL data??

[sasiparupudi1]

May be drop unnecessary columns like large string fields

also, as peter suggested ,drop tables immediately after the concatenation

[bhaskar_sm]

Hi,

Are you reading those tables from QVDs ?? if no then try to store all those tables in QVD and then perform the staging.

which is noting but combining Qvds in Group like in first you can concatenate and store 4 set of table in one Qvds and likewise do for reaming tables, so on in next you read all stored Qvds and combined into one final Qvds.

NOTE: please drop the table after Store command.

i hope by doing this i will take less time.

if you found this helpful, please mark helpful or correct.

Regards,

Bhasker

[Peter_Cammaert]

There you have it. If I understand you correctly, the STORE consumes almost all RAM (7.7 or 7.8 GB with some chunks left for Windows itself = 100%) and as a direct result will become extremely slow (swapping, paging, etc)

You can either 1) Add more RAM (always the best and - depending on your region - cheapest solution if the data amount is there to stay) or 2) Fragment your table into different QVD chunks (for example one file for one year) or 3) optimise the content of your table before STOREing it.

Best,

Peter