Data Flow control mechanisms in Talend?

kleinmat — Wed, 16 Jul 2014 10:41:37 GMT

Hi everyone,
I have a job that implements a simple web service.
The job should do this:
1. Retrieve some of the request parameters and memorize them
2. Use those parameters in a SQL query (its where-clauses) to fetch data
3. Filter that data
4. Create some response
I have problems with filtering in Talend, though; here's a sample of the process that needs to happen:
TAKE all the data you fetched from the database
SPLIT the result set into 2 lists:
List A contains all result set records that match a certain pattern (that have certain values in a certain field)
List B contains the rest
IF List A is empty, use List B
IF the List (A or B - depending on the previous step) has more than 20 records, throw a fault
SORT that List by the Date in field A or field B (Take field B if field A is empty)
Take the top 1 record
If field C of that top 1 record contains a certain value, throw a fault
Take field D and create a response
That's a simple filter algorithm that requires not only data flow (sorting, simple filtering) but also flow control (IF/THEN-conditions, branches, counting of number of rows etc.)
But how can that be done in Talend?
I find the filter and row sort mechanisms too rudimentary (or I don't understand them fully).
And I have not seen anything that implements conditions in an intuitive way.
Any ideas?
Thanks
Matt

Re: Data Flow control mechanisms in Talend?

Anonymous — Wed, 16 Jul 2014 11:08:57 GMT

Hi Matt,
TAKE all the data you fetched from the database
>> take database Input component
SPLIT the result set into 2 lists:
>> Use tReplicate component
List A contains all result set records that match a certain pattern (that have certain values in a certain field)
>> Use tFilter with appropriate condition for A output connect output to tHash component
List B contains the rest
>> use tFilter with another condition for B output connect output to tHash component
IF List A is empty, use List B
>> Use RunIf clause from tHash component use appropriate condition
IF the List (A or B - depending on the previous step) has more than 20 records, throw a fault
>> Use RunIf clause from tFilter use appropriate condition and connect to tWarn/tDie or generate your exception using tJava
SORT that List by the Date in field A or field B (Take field B if field A is empty)
>> Use RunIf and take appropriate tHash component and get the records
Take the top 1 record
>> Need to use again tFilter to get the Top record based on the data values
If field C of that top 1 record contains a certain value, throw a fault
Take field D and create a response
>> Use context variables during your job for conditions to be used in RunIf ...
Hope you got some idea...
Thanks
vaibhav

topic Data Flow control mechanisms in Talend? in Talend Studio

Data Flow control mechanisms in Talend?

Re: Data Flow control mechanisms in Talend?