Limit 30,000 rows: our response

[Anonymous]

We get this question/feedback a lot: why does the tool limit to the top 30,000 rows, or: 30,000 is not enough.
Data Prep Free Desktop loads the entire dataset in memory. 30K is not a hard limit, just a safeguard to stay beyond acceptable response times for the average hardware. As more high-end hardware can handle more rows, and because 30K may be too little for a file or too many for another, in an upcoming upgrade there will be a UI control to let you increase this limit as you see fit.
In the meantime you can play trial & error by changing this arbitrary limit in a config file located here on Windows: \config\application.properties. Just edit the number in your favorite text editor. Sorry Apple users (including yours truly) the similar file on OS X is not as easily editable.
The commercial add-on due in June will feature more sophisticated techniques and scale with large files.

[Anonymous]

Where exactly is this file located? I can't seem to find it.

[Anonymous]

Where exactly is the file located? File name and path

[Anonymous]

As mentioned in the first post, the file name is application.properties. It is located in the config folder in the Data Preparation installation folder. If you used the default installation path, then the path is C:\Program Files (x86)\Talend\Talend Data Preparation Free Desktop 2.1\Talend-DataPreparation-Free-Desktop-windows-2.1.1\config\application.properties

Regards,

Gwendal

[Anonymous]

Hey,

I ran into some of the issue discussed above. 

Had to restart the service using:

Program Files (x86)\Talend\Talend Data Preparation Free Desktop 2.5\Talend-DataPreparation-Free-Desktop-windows-2.5.1/stop.bat

And running stop.bat as an admin, that did the trick.

I set my limit to 5,000,000 and was unable to load a file with 3.2M records.

Then failed again with a 250k record.

It either crashes or reloads the Data Sets screen without my file in it.

Has anyone hit a similar limit or run into these issues?

[Anonymous]

Must say, as a (currently) non-user I'm still not clear:

How does one process a file that is, for example 500,000 rows? I take it this simply is not possible at all with Data Prep free version?

I don't understand how the paid version works: presumably the 10,000 row limit is a 'working' limit but the whole file is still processed at the end? Or not?

And if not, how exactly is one supposed to process large files?

Thanks in advance,

Chris

[smallet]

Hello,

Processing more than 30,000 rows is not possible with Data Prep free version without changing the parameter define above.

With the paid version, this is how it works:

• the sample loaded in the grid (in the web ui), is limited to 10,000 rows
• you can use "Fetch more" option in combinaison with filters to work on records after this limit
• you can export "All data", this will process all the records, not only the displayed sample

Hope it helps