topic Stitching a Spark Component to Its Source and Destination in Talend Data Catalog

Stitching a Spark Component to Its Source and Destination

AbdullahMastan — Wed, 14 May 2025 07:11:02 GMT

Greetings,

Problem: Stitching a Spark component containing Python scripts to its source and destination (PostgreSQL).

What I have tried:

Created a Spark component and imported its scripts, but I am unable to stitch it to its source and destination.
Created a Data Mapper and manually defined sources, destinations, and transformations. However, this approach is impractical for companies relying on large ELT scripts.

Given the massive Python codebase we have, is there a way to automatically discover the transformations a dataset field(s) has undergone within the script?

What would be a more effective approach to handle this scenario?

#Talend Data Catalog

Re: Stitching a Spark Component to Its Source and Destination

AbdullahMastan — Mon, 19 May 2025 03:27:04 GMT

Hey @Shicong_Hong any comments, or do you know who could help?

Re: Stitching a Spark Component to Its Source and Destination

AbdullahMastan — Tue, 20 May 2025 04:55:45 GMT

The replies from the community on this post are not visible on any of my device.

All the more, my reply to someone else's reply has also disappeared.

What's going on?