Hello,
I have been wondering if it could be possible to have dynamic metadata in Talend jobs and came to the conclusion it wasn't (I read, among other posts, 1934)
Let me explain a little.
A use case would be to copy all tables from given a database (let's say a an MSSQL one) to a new one (let's say a MySQL one). The purpose would be to do an easy data migration in a single job run with different parameters (i.e. the table name to copy).
Would this kind of thing be possible ?
As a matter of facts, I have been a DataStage Enterprise Edition (it was called PX at that time) user. And the solution for that kind of problem, if I remember well, would have been to use RCP : Runtime Column Propagation (may there would be problems because of a lack of database connectors, which is not a problem for Talend). Indeed, in DataStage components (call stages), there was a checkbox called "RCP" which meant : "may be you specify metadata in the stage (but it's not compulsory) but may be other columns exist and they will be propagated at runtime". Those columns were implicitly propagated at run-time until the RCP checkbox was unchecked in a stage.
So, to use this capacity, it was also possible to make text-based schema (and use them as a parameter) or, if I remember well, use a "select * from table" in a database input stage which could generate the metadata schema at runtime.
So is this kind of thing possible in Talend (I suppose it isn't, if I understood well)? Is it planed ? Are there legal issues (patent?) which could prevent you to do that kind of thing?
BTW I hope I am allowed to write about DataStage in that forum ...
Well, for me Talend is really a great product, but this point is the only one (may with parallelism) which make it not as good, for particular usage, as DataStage PX.
Dynamic Schemas is a new feature supported in Talend 4.1. This means that you can create migration processes without defining (or knowing) the full data schema of the sources.
This post seems to be have 'died' a long time ago, but I figured I'd post here incase anyone at any point (ever) sees this and has a solution. I have also posted my own forum but with no responses.
Basically, as the original post says, parallelism can be a problem with Talend (I use GreenPlum) and though it works technically, there seems to be no way of creating a table with a distribution key (for GP) and WITHOUT a primary key (which is PSQL based). Anyone found a way of getting this to work?
I saw someone else in the thread commented on this too.. I'm just wondering if anyone knows of a solution that I seem to have missed (repeatedly).
