@rhall wrote:
This isn't going to be possible as you describe it, but there are potential workarounds depending on the variety of schemas and how much work you want to put in. However, the first thing I want to point out is that you cannot write one job to deal with any schema. A common request in here is for a "magic job" that will dynamically work out the schema, understand the changing mapping requirements and write to a dynamic table/file. If this sort of job was easy, it would largely do Data Integration people out of a job (....this won't happen in our lifetimes since the permutations of scenarios that this magic job would have to handle are practically endless).
"Magic Job" rant aside, there are possibilities.
1) You could build a parent job which selects a suitable child job to pass the file to depending on data related to the file. This is probably the easiest approach to this problem, but will take a reasonable amount of work building several child jobs. It also requires that you know of the possible schemas at design time.
2) This method is similar to the above, but instead of nesting your child jobs inside the parent job (or controlling job) you can configure your framework so that all are separate jobs. The parent job can easily start child jobs in separate JVMs. This would enable an easily extendable way of achieving this. You do not need to know of all of the schemas at initial design time and if a new one comes along you can simply create a new job for it. So long as the parent job has been created to be a dynamic as possible with how it starts "child jobs", you could incrementally build up quite a solution using this.
3) If you are comfortable using Java you *could* build a process which is Java heavy and dynamically works out the schema. This would require a fair amount of going under the bonnet and seeing how Talend jobs work. I have done similar things in the past and am pretty sure your requirement could be achieved, but this may be approaching a solution which is far too Java heavy for a lot of people. Basically you can do whatever you want in Talend with Java. You have access to a world of Java APIs and can implement practically anything using them.
There are probably other ways that you could approach this and I would need to know the complete requirement before I would personally decide upon the one I would use. But I should point out why this is not possible without a lot of work. If you take a look at the Talend code (look at the code tab on the development workspace) you will see that the schemas are implemented in code ( which is compiled). Take a look for static classes called "row{number}Struct" to get an idea. Since the configuration of components and jobs relies heavily on schemas, they are essentially the backbone of Talend jobs. If you use a repository schema in a series of jobs and modify that repository schema, every job using it will need to be altered to safely incorporate that change. More often than not, this is pretty automatic. The Enterprise Edition has some components that are built to enable more dynamic schema functionality without the need for long winded workarounds, but you would still be hard pushed to enable your scenario without a fair amount of work.
Thanks for the response @rhall.
I was actually busy with your first suggestion when I decided to ask the community.
It looks like I'll have to extend my timelines to focus on going "under the hood".
Do you mind sharing some ideas on what you did to solve similar problems as you mentioned, just to shed some light for direction.
To give you an idea of what I'm trying to solve:
I have a tHive input and tSybase input going into a tMap for a join which outputs some file, say tFileOutputDelimited.
I want to be able to use a parameter for the schema on the 2 inputs, parameter for the tmap and the output file.
I'd appreciate just a point in the right direction to avoid barking at a lot of wrong trees.
Thanks again
