Error: Out of Virtual and/or Logical Memory, allocating 364.8 MB

[shumailh]

I am trying to load data from the text file but i couldnt load it completely as i am getting error after record number about 185,000 whereas the full text file contains about 480,000 records.

Below is the error message i am getting. Can any one please help me in this?

[shumailh]

Below is my QlikView Setting for your reference,

[rwunderlich]

Can you post your load script and a couple sample lines of data?

-Rob

[Oleg_Troyansky]

185,000 record - doesn't sound too big... Since your text file load is not finished, I don't think that it's caused by synthetic keys.

1. How many columns do you have in your text fie?

2. What is the total memory on your computer?

3. Check Windows "Tas Manager" - do you have any available memory at the time of the crash?

Oleg

[shumailh]

Hi Oleg,

Yes, I also wondering why it's generate this error 185,000 is nothing as i have loaded huge records.... Below are my specifications.

1. How many columns do you have in your text fie?

Ans: 129 Columns

2. What is the total memory on your computer?

4 GB

3. Check Windows "Tas Manager" - do you have any available memory at the time of the crash?

yes only 1 GB is in use and 3 GB still free but i dont know how come it generates this error..

Hi Rob,

my script is simple as under,

LOAD CHPH_REC,
CARD_NO,
FILLER1_PH,
FIRST_NAME,
MIDDL_NAME,
LAST_NAME,
NATIONALITY,
GENDER,
DOB,
MARITAL_STATUS,
MOTHER_MAIDEN,
SPOUSE_NAME,
EMPLOYMNT_STAT,
PH_MONTHLY_SALARY,
CH_LOS,
DSA_NAME,
SOURCE_APPLICATION,
EMPLOYER_NAME,
STAFF_NUMBER,
OFFIC_TEL,
MOBILE_NUMBER,
RESIDENCE_TEL,
STATEMENT_ADDR1,
STATEMENT_ADDR2,
STATEMENT_ADDR3,
PH_CARD_INDIC,
PH_MAIN_CARDNO,
PH_PRODUCT,
AMED_USER_2,
PASSPORT,
CARD_SCHEME,
LANGUAGE_INDIC,
BASE_SEGMENT_NBR,
MEMO_2,
NA_COMAKER_NAME,
EMPLOYEE_CODE,
USER_CODE_3,
INS_PRODUCT_PH,
INS_STATUS,
PROMO,
EMAIL,
EMAIL_FLAG,
CH_REC,
CRL_CARDNO,
EMBOSSING_NAME,
CREDIT_LIMIT,
BILL_CURRENCY,
AUTH_STS,
DELQ_DAYS,
FILLER_CH1,
VALID_FRMDAT,
EXPIRY_DATE,
BRANCH_NUMBER,
BANK_ACCT_NUMBER,
PAYMENT_METHOD,
PAYMENT_PERCENT,
ACCT_CREATION_DATE,
FILLER_CH2,
SERVICE_CODE,
PVV_OFFSET,
CVV_CVC,
ECOD_DTL1,
ECOD_DTL2,
ECOD_DTL3,
LST_ACTIVITY_DATE,
LST_STMT_ISS_DATE,
CH_CARD_INDIC,
CH_MAIN_CARDNO,
PREV_CARD_NUMBER,
CUR_BAL,
LST_STMT_CBAL,
LST_STMT_GEN_DATE,
LST_STMT_MIN_DUE,
PAST_DUE_AMT,
TOTAL_DUE,
DATE_LST_PAID,
AMOUNT_LAST_PAID,
DATE_PMT_DUE,
DATE_LST_CRLMT_INCREASE,
CH_MONTHLY_SALARY,
CASH_UTIL_PERCENT,
CURR_CYCLE_DUE,
DELQ_HISTORY,
HIGH_DELQ_BUCKET,
NEXT_STMT_DATE,
CARD_RENEW_DATE,
LST_REISSUE_DATE,
CH_PRODUCT,
WAIVE_LATE_CHG,
WAIVE_INTR_CHG,
WAIVE_SVC_CHG,
WAIVE_FEE_CHG,
WAIVE_LATE_NOTC,
WAIVE_OVLM_NOTC,
WAIVE_OVLM,
WAIVE_NSF_FEE_IND,
FILLER3,
CREDIT_SHIELD,
INT_STATUS,
BLOCKCODE1,
BLOCKCODE2,
BLOCKCODE3,
CTD_RTL_PURC,
CTD_RTL_CASH,
TOTAL_CREDIT_LIMIT,
OPEN_TO_BUY,
CURR_FIRST_USAGE_FLAG,
PRIOR_FIRST_USAGE_FLAG,
DATE_EXPIRY_ED,
DATE_LAST_EXPIRY,
TOTAL_BALANCE,
ACCT_STUP_FLAG,
ED_CRLIM,
ED_OPEN_TO_BUY,
FILLER4,
BLOCK_DATE1,
BLOCK_DATE2,
BLOCK_DATE3,
ED_EMB_DATE,
ED_ACTV_DATE,
INS_PRODUCT_CH,
SECURE_WALLET,
CRD_HLDR_FLAG,
V1,
V2,
V3,
V4,
V5,
ACTIVE
FROM D:\CARDS\Final\ABCCHB31.txt (ansi, txt, delimiter is '^', embedded labels, msq);

Attached is the text file FYR,

[shumailh]

SYStem Configuration

[Oleg_Troyansky]

looks very strange... Last question - are you using QV Developer (Enterprize?) Check your license. The reason buing - Professional license could apply a limitation on the number of records that you can pull. However, that limit was around 65K, I think...

A can't see any other reason...

Oleg

[rwunderlich]

It could be just due to a lot of data that won't fit in 32bit. You appear to have a lot of uniqness in your data. Any idea how long an average input row is?

Try this to see if you can get more rows loaded. Use typing functions like num() on the fields where it makes sense. For example,

num(CUR_BAL) as CUR_BAL
date(DOB) as DOB

You may get more efficient storage that way. Let me know if it helps.

-Rob

[shumailh]

I am working on Developer Full and Licensed Version and I got the license recently for the QlikView 8.506.... By the way i was parsing more than 1 billion records previously in different project on the same PC and version.

I couldn't understand where this limitation applies even the memory space on task manager performance tab is almost 75% available as i told u.

Is there any other configuration or setting for Qlik?

[shumailh]

Rob,

For this project, I have tried two different ways as below

Way 1: http://community.qlik.com/forums/p/17422/68208.aspx#68208

In this way all the 480k records successfully parse from the RAW source file but the problem is that i am not getting the exact numbers for the V1, V2, V3, V4, V5 and active.

ABCCHB.Dat -----------> Load DAT.qvw
RAW File

Way 2: Current

I Parse the RAW file using VB Script (Generate Text File.vbs) which is attached. This VB script parser generates the data with the delimitation ^ and make 2 rows in to one and apply all the conditions for field V1, V2, V3, V4, V5 and active and generate it.

I verify the data and conditions used in this text file on SQL Server and the numbers are matching for field V1, V2, V4, V5 and active.

data in both the source file is same i.e. about 480,000.

i am getting the correct number in 2nd way but not able to completely load all the raws...

I think may be you can resolve the way 1 if will see the VBSCRIPT File attached.

ABCCHB.Dat -----------> ABCCHB.txt -----------------> ABCCHB.txt -----------------> Load text.qvw

RAW Source VBSCRIPT File with Delimitation QlikView File