Hi everyone,
I am having a bit of a problem exporting a large amount of data (~6million rows) from a pivot table in QV to tab delimited files. Using the built in export function to export the whole table is far too slow and often just seems to lock up QV on the server.
I have read suggestions on here for similar problems to STORE the data on reload. Unfortunately this is not an option as we have 8.5 and it does not seem to support exporting to tab delimited .txt (only exporting to the proprietary .qvd format) and the server that this would be processed on does not have ODBC drivers that I could access to do this.
The solution we thought of originally was to create a list box with all possible selections from the most evenly distributed field and looping through each one, selecting the table where the field matches that value, then using the inbuilt export function on the resulting table into a seperate text file. With smaller amounts of data this increased the speed of the process immensely (we assumed this was due to better memory read/write efficiency than exporting one single file). Unfortunately when trying this with larger datasets the distribution of data became more uneven and eventually certain selections were too big to export.
I tried to do a stacked selection loop eg. every Field B in Field A is selected but the data distribution is still too uneven to work and the more you stack the selections obviously the worse the speed of the process is.
Currently I am exploring making the selection listbox out of different expressions to search for eg. a list of all possible last 3 characters and selecting items in the field using that and wild cards like "*AA".
Ideally I want to simily select rows 1 to 50000, export, select rows 50001 to 100000, export etc. Seems pretty simple, but I can't find anything in the API that allows me to just select a certain set of rows.
Does anyone have any suggestions of how I might be able to achieve this?
