First, an apology to the group if this topic has been addressed; please direct me to that topic.
(The subject line would not hold the entire subject: "How to create multiple output records from a single input record; how to create one output record from multiple input records")
I am looking at Talend Open Studio for Data Integration, version 5.5.
We process data from many clients, and have no control over the client input. We need to generate data which populates several (at least 2, perhaps 20+) data records (each with a unique format) which will later be parsed and loaded into multiple tables in a database. Each client's data is reformulated into a "style" which can be processed within a common system.
Data may come in as XML, with extensive <xsd:complexType> records and sub-records; this data is collected across several records within a single transaction. (In this discussion, a "transaction" is a month of data all tied to a single bill / statement.) Data might also come in as CSV, or "standard COBOL", or generic fixed field. In almost all cases, each transaction consists of multiple input records with varying formats: Master, payments, adjustments, claims, etc.; each with a unique format.
In some cases an input record will contain a table; and each tabular entry might or might not require an output record to be written; so decision logic, and one-to-many is required.
The output records will be of several types (each type destined for a unique table) and will not have a single standard format; the formats will be application-specific, and so must be custom-constructed for each application. The output records are constructed so that all the data records for a transaction can be gathered into a complete image of the transaction.
And so, the question is, "how is an Open Studio Project configured to intake multiple input formats and output multiple outputs in a single file? (Output to multiple files, one per format, might be acceptable if the remainder of the problem can be solved.) The sequence of the output records is not significant, since there will be keys in each output record used to reassemble each transaction. (Multiple passes through the input data, generating multiple output files, would probably be highly inefficient; and makes for very complex relationships in building a consistent set of keys across transactions.)
I need insight (and, best case, examples or a tutorial) which I can use as a model.
In advance, Thanks All.
And so, the question is, "how is an Open Studio Project configured to intake multiple input formats and output multiple outputs in a single file? (Output to multiple files, one per format, might be acceptable if the remainder of the problem can be solved.) The sequence of the output records is not significant, since there will be keys in each output record used to reassemble each transaction. (Multiple passes through the input data, generating multiple output files, would probably be highly inefficient; and makes for very complex relationships in building a consistent set of keys across transactions.)
Would you mind
elaborating your case with an example with input and expected output values? Are you looking for TalendHelpCenter:tMap which transforms and routes data from single or multiple sources to single or multiple destinations.
Best regards
Sabrina
