Making a dynamic Schema repository

[pthomas]

I have several components that depend on a schema and most I manually create based on what I'm working with.
I use the data to create a file then call a job that loads that file into our storage.

How can I use a context variable in the load job to call a Generic schema?

The issue is that every job in this set of flows needs to call a load job. I can paramertize the connection, bucket and file name. But the schemas will be different. I can create Generic schemas but have no way to call them dynamically. If I could, then I would only have the 1 load job.

Below is a job that I'd like to be a 1 job loads all.

[gouravdubey5 · Accepted solution]

Hello,

In Talend Studio, a Dynamic schema can be stored and reused from the Repository, provided the metadata itself is defined as Dynamic.

Key points:

The repository metadata must contain a single column of type Dynamic.

This Dynamic metadata can then be dragged into jobs and reused across components.

The dynamic structure is resolved at runtime based on the incoming data.

Notes:

Dynamic schemas must be defined in the Repository; they should not be created manually in job schemas.

Ensure that all downstream components support Dynamic input.

Dynamic schemas are intended for flexible structures, not fixed column mappings.

Best practice:
Centralize the Dynamic metadata in the Repository and reuse it consistently across jobs to simplify maintenance.

Thanks,

Gourav

[pthomas]

Interesting.  I see the Dynamic data type now I just need to get in there and make it work.  This makes so much more sense than the 12-15 jobs that all do the same thing but with a different schema that I am using.

[gouravdubey5]

Hello,

Thank you for the update finally you got the datatype 🙂 Happy to help!

Thanks,

Gourav