Hi guys,
it does take a bit of time to open it. |
Please can you tell us :
- how much time it opens,
- how many tables
- how many and columns
- with which OS
- with which TOS version
1. Make tMap sort aware. So if lookup is sorted on join keys and if tMap knows it, it can do the joins more efficiently as it does not have to go to the end of the table each time. |
In memory mode, data are hashed to be retrieved quickly.
In disk mode, data are sorted on the result of the keys expressions.
The next release 3.1.0 will allow to reload the lookup at each row, then you will be able to extract only the desired data of a database table. The 'Store on disk' mode will remain useful when files from data will have to be joined.
2. Make it able to handle large data sets easier. I've had errors when I had a lookup table as big as 200K rows. I got some NetBeanLookup error (I do not remember exact error). It was almost like an Out of Memory error. I was able to get around by making the table as skinny as possible. This was when I was caching it on disk (not in-memory). This is critical. |
I advice you to change the advanced property called "Max buffer size", set it to a value around 200 000 rows by buffer, you have to set this value according different parameters. It should resolve your OutOfMemory problem definitely.
3. A scroll bar for large tables inside will be great. |
Indeed it seems not possible to change the column size under Windows (it is possible to fix it), yet don't forget you can use the tab "Expression editor" at bottom or the "Expression builder" in columns.
Ability to directly lookup DB tables. If we have a million row table (which is not even that big), why spool all the rows to disk? Based on my observation, TOS is going to fail (at least in Java). Why not add ability to directly connect to the database table and do a lookup? Do we have this functionality in some other component? |
As described it will be possible to do it into the next release 3.1.0 M2, and more possibilities like Volker said it using a global var in any fields as you want.
For example you will be able to reload the lookup with a different file, or change the WHERE conditions at each main (or above lookup) row.
I uploaded a screenshot which shows an example of use, the UI is not definitive.