topic Re: Database queries spread over hundred Jobs . Any tip to extract statements ? in Talend Studio

Database queries spread over hundred Jobs . Any tip to extract statements ?

Anonymous — Mon, 15 Apr 2019 19:23:03 GMT

I'm newbie with Talend and working on a install with hundred jobs with many database queries spread over all jobs developed by many without control .I'm affraid some queries are same and duplicated .

Question : There are any way to extract these queries statement inside Talend in a easy fashion ? Any Talend component that can help me ? I'm not willing to examine tons of Java code to extract it. Thanks!

Re: Database queries spread over hundred Jobs . Any tip to extract statements ?

Anonymous — Tue, 16 Apr 2019 11:00:23 GMT

This can be done quite easily by looking at the backend files. All information in Talend jobs is stored in .item and .properties files in your Workspace folder. These files are actually XML files, so you can process their information quite easily. This is NOT a supported process and you WILL cause a lot of damage to your project if you modify these files. So to do this, you will need to take a copy of this folder structure, move it somewhere safe and analyse the copied .item files. You can actually build a Talend job to do this quite easily. Using a tFileList component you can navigate through your folder structure, then all you need to do is to work out the structure of the XML and identify how to find SQL queries. Once you have that, it is pretty straight forward.

On another note, why is this a concern? Is it because you think that there may be several jobs doing the same thing? Or is this simply because you do not want the same query fired multiple times? The reason I ask is that it is really not unusual to have the same queries fired multiple times by multiple jobs depending on how your project has been built. Each job is an independent Java application in its own right.

Re: Database queries spread over hundred Jobs . Any tip to extract statements ?

Anonymous — Tue, 16 Apr 2019 21:50:57 GMT

I believe there are several jobs with the same query and I would like to centralize those queries under a database package to provide a centralized source and some control over it . Will follow your tips . Thanks a lot!

Re: Database queries spread over hundred Jobs . Any tip to extract statements ?

Anonymous — Tue, 16 Apr 2019 22:36:57 GMT

Fair enough. Seems like a reasonable thing to do.

Remember to work on a backup of your workspace 🙂