.TXT file upload interrupted, loads to application incomplete
Hi all, I have a very odd issue with my data in Qlik.
I've been trying to import a text file, 2GB in size, from my local drive into Qlik (to put it through the whole data, extract, standard process). But when I import the cleansed, standardised file into my application, it's only 54MB. It's missing most of the data.
I've tried it on a completely new Qlik file, just with the import code and nothing else; no 'where' clauses, nothing. But it still only loads a small amount of the data (1,021,413 records).
I mean, I'm fairly sure about how many rows are loaded into the application. I put a count on the primary key of the table, and every time I load it, only 1,021,413 rows are loaded (out of about 8 million).
When you say try the same thing in DB, do you mean my DataBase outside of Qlik?
I don't think it would be possible I'm afraid, as it's 2GB in size, and it contains a lot of business-sensitive information (all the competitor information of a large company)
But I've had a look at the DataBase that it comes from (SAS Enterprise Guide query). There is nothing particularly different about the record that it stops at. I've read somewhere that special characters can interrupt the load, but I see none on this record. However, since the txt file is based on a webscrape, i.e. the data is taken directly from websites and dumped into a table, do you think the formatting of the data could be affecting the load?
I've tried something similar. There's a field called retailer, the name of the competitor. Currently, it's only pulling in the top 1 million or so rows, so only the first retailer is in the data (Screwfix). I've tried putting in a where clause:
LOAD * FROM <Table> WHERE Retailer = 'Selco'
I know that there are records in that txt with Selco as the retailer, but adding this clause results in 0 rows being fetched.