Error: Out of Virtual and/or Logical Memory, allocating 364.8 MB

[shumailh]

I am trying to load data from the text file but i couldnt load it completely as i am getting error after record number about 185,000 whereas the full text file contains about 480,000 records.

Below is the error message i am getting. Can any one please help me in this?

[rwunderlich]

With regards to the overall problem, I understand from your earlier post that you are trying to replicate the SQl Server views. Is there some existing process that loads these text files into SQL Server so you could extract data to QV using the SQL views? You indicated that youe were comparing the numbers from your load with the SQL Server numbers.

Hard to understand exactly what's going on without seeing the data, which appears to be financial data so I understand you can't post.

1. What are the file sizes (in kb) of ABCCHB.Dat and ABCCHB.txt ?

2. Have you tried adding NUM() and other formatting to the load as I suggested earlier?

3. Can you do a limited load of 100 rows, scramble the document and upload here so we can see it?

-Rob

[shumailh]

Actually, I cannot use SQL Server because I dont have SQL Server on Production and I am working on UAT Server where I am using SQL Server for other projects even though I cannot integrate this SQL Server instance to the Production. So I need to develop this project standalone on QlikView. whereas In SQL Server I was using DTS to load and transform data just to reconcile numbers.

1. What are the file sizes (in kb) of ABCCHB.Dat and ABCCHB.txt ?

Ans: ABCCHB.Dat = 600 MB after parsing and Transformation ABCCHB.txt become 450 MB

2. Have you tried adding NUM() and other formatting to the load as I suggested earlier?

Ans: I tried this technique but still i was getting the same problem.

In technique 1 which I have sent you on earlier post, some of the fields are not correctly transform and parse. I dont know why even though the delimitation code for vbs file in Technique 2 and for Qvw in Technique 1 is same.

[rwunderlich]

Can you do a limited load of 100 rows, scramble the document and upload here so we can see it?

-Rob

[shumailh]

Attached is the load for your reference, I was trying to send you the full text file but i couldnt attached it on the forum

[rwunderlich]

In your file all the data is stored as strings, which occupies quite a bit of RAM. In particular the CARDNO fields, which are 19 bytes long but in the sample always have leading zeros. I have trouble getting QV to convert a 19 byte string to numeric -- it seems to be limited to 14 butes input. Would it be acceptable to drop the leading 5 zeros from the CARDNO fields so they can be made numbers?

Attached is a modified script that makes the CARDNO and a few other fields numeric, which should reduce the RAM required for load. Can you try it with your data to see if more rows will load?

-Rob

[#7]

Go to Setting, then User Preference or Ctrl+Alt+U, Select Object Tab, then Uncheck "Ignore Control Limit".