I load from a 2.2 gb .csv file with about 22 million rows and ~10 columns everyday on my laptop. It takes about 10-12 minutes for it to go through the file and as long as I restrict how many rows I load with a WHERE clause or a GROUP BY clause (<6 million is usually safe), I don't get out of memory errors. However, I recently made a .qvd with about 8 million rows (I think, the qvd is like 660 mb) on a better computer that I'm trying to load on my laptop. The problem is that on the .qvd, my laptop can't even come close to even looking through the whole file without getting out of memory errors every single time.
So my laptop can sum 22 million rows by date without getting out of memory errors but it won't load like 200,000 rows from the .qvd? Can anyone explain this?
And I just tried this:
TransactionsTemp:
FIRST 300000 LOAD
sender,
receiver,
tx_set,
date,
time,
sender_transaction_id,
originating_transaction_id,
//sender_entity_type,
esiid,
customer_name,
reject_flag,
reject_reason_code,
reject_reason,
left(sender_transaction_id,3) as 'service_order_type'
FROM
(qvd)
WHERE esiid = '15981561258'; //one customer
which loads like 50 rows but has to look through the entire file and it worked fine in like 1 minute....
what's going on? If i took out the first 300000 and tried to put WHERE clause that would load like 500,000 rows it wouldn't work for some reason but I can 6 million fine and I can obviously do things that require going through the whole file just fine.
Thanks in advance.
