topic Re: Where clause on Talend in Talend Studio

Where clause on Talend

Anonymous — Sat, 16 Nov 2024 14:22:38 GMT

I extract data from a big DataWarehouse Oracle.
Currently, I work on a table which contains about 10 000 000 rows. So I need to filter this data. In Sql I use the where clause but, with Talend I want use tFilterRow. I see a big problem :
when I transmit my filters in SQL with the where clause it's the server which stocks Oracle which manages the filters. But when I use tFilterRow, in fact all the rows are extract and they are filtered in "local". The exection is very different (30 seconds versus 3 hours...)
It's necessary to give my filter in SQL at Oracle or I need to param something ?

Re: Where clause on Talend

Anonymous — Tue, 13 May 2008 16:50:54 GMT

Hello,
Said ad-hoc (I´m a newbie to TOS):
1.- Customizing application env. (like increasing JVM´s Heap for such a volume of data)
2.- For TOS developer: ?Buffering data within Java routines, which handle high volume data, whereby code sets the size of buffer in dependency of configured heap size, set by user ?.
Isn´t a good practice to move the TOS toward easy processing of Higher Requirements?
aski

Re: Where clause on Talend

Anonymous — Wed, 14 May 2008 16:49:08 GMT

Hello,
I would really do the where in the SQL query.
Regards,

Re: Where clause on Talend

Anonymous — Wed, 14 May 2008 17:05:54 GMT

Hi mhirt,
Sorry but I don't understood your answer.
You want say that improve SQL builder (particulary where clause) for the next version of Talend is in prevision or you want say that I need to develop the component ?
Thx for your answer.

Re: Where clause on Talend

Anonymous — Wed, 14 May 2008 17:58:23 GMT

If you ask tOracleInput to read 10 000 000 rows, in all the cases, it will be very long.
Simply add your filter to your sql request in tOracleInput.
Why are you speaking about "Improving SQLBuilder" ?

Re: Where clause on Talend

Anonymous — Thu, 15 May 2008 08:33:05 GMT

For me,
an ETL like Talend is very interisting because the user need to param your soft by a GUI. But in SQL Builder, for the where clause you need to add manually the where condition. For me, I think it will be very interisting if SQL Builder purpose the tFilterRow component in order to create the SQL request with where clause.
Do you see what I want say ?

Re: Where clause on Talend

Anonymous — Thu, 15 May 2008 21:16:54 GMT

You mean adding filters in SQBuilder configurable like the filters of SQLBuilder ?
It can be an interesting feature.
Do you mind creating a feature request on our bugtracker ?
Regards,