We get this question/feedback a lot: why does the tool limit to the top 30,000 rows, or: 30,000 is not enough.
Data Prep Free Desktop loads the entire dataset in memory. 30K is not a hard limit, just a safeguard to stay beyond acceptable response times for the average hardware. As more high-end hardware can handle more rows, and because 30K may be too little for a file or too many for another, in an upcoming upgrade there will be a UI control to let you increase this limit as you see fit.
In the meantime you can play trial & error by changing this arbitrary limit in a config file located here on Windows: \config\application.properties. Just edit the number in your favorite text editor. Sorry Apple users (including yours truly) the similar file on OS X is not as easily editable.
The commercial add-on due in June will feature more sophisticated techniques and scale with large files.
Downloaded Data Prep and installed. Install went well. First Excel file was 77,000 rows. Loaded 30,000. Found this message and implemented changes as 300,000 then at 3,000,000. Data Prep shows 30,000/30,000 in the top corner. Using Data Prep version 1.3.0. Tried closing all browsers, deleting the loaded files and reloading, deleting the preparation. At a loss as to what to try next.
Restarted computer and the issue was resolved. Appears there is a background process that reads the config file once per computer start.
Hi Kyle, I am also stuck with same issue. We have configured a job in talend studio which passes csv file that holds more than 70000 records using tFileInputDelimited component to tDataPrep component, But it goes in infinite loop and even with blank recipe and nothing is coming as an output. Is there any workaround to divide csv file in multiple files?
What is the cost of the commercial version of Data Preparation and Data Quality tools. How do I get the commercial version.
If you have a matrix for pricing & features for all your tools it would help
Could you please send an email to firstname.lastname@example.org with your requirement? Our colleagues from sale team will assistant you to optimize product pricing.
Feel free to let us know if it is Ok with you.
I have version 1.1 and have edited the file mentioned above and I still am restricted to 30,000 records. How an I remove this limit? It this is the limit I will need to do my profiling directly in excel.
rugged hand held computers