High level project design, data studio, managing complexity

[Anonymous]

I'm trying to integrate 10 disparate sources of CSV data into a database, which could be a source for an integration.

The problem is that the source data will need cleansing, and in some cases standardization.

I know how to do each task on its own, but the complexity is becoming a concern. How would u break up the jobs? How would you have it come back together?

[Anonymous]

Hi,

    Why don't you create each functionality in multiple sub jobs and call them from a parent job using tRunJob Component?

    In this way, the individual functionalities will be preserved in each job but still you can maintain the overall flow.

https://help.talend.com/reader/jomWd_GKqAmTZviwG_oxHQ/ICv6hRE2pgpUtQFvQDmxkg

    Please refer the above link for tRunJob properties along with sample scenarios. This detail will guide you to resolve your query.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

That is a really good idea for once I get to how to execute the jobs in the project, but I'm at the design phase.   How would you organize this so I get some degree of reuse for common stuff between jobs?  How would you make this maintainable?

[Anonymous]

Hi,

    What I usually do is to mimic the SoA type of Architecture. This means I would take an epic and divide them to standalone jobs which can cover a full service. Within each service, you can add multiple subjobs which will each subject area of a service.

   A simple example of a Job is Customer Creation and sub jobs which needs to be identified is Addess Loading, Emil Loading, Telephone Loading, Adding Foreign keys of these tables back to Customer table etc.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂