topic Migration of datastage code to Talend in Talend Studio

Migration of datastage code to Talend

Anonymous — Fri, 24 Aug 2018 15:34:14 GMT

Hi,

My team is planning to move on to Talend from Datastage. We are in initial stage right now, were just trying to figure out if we can migrate Datastage code directly to Talend.

Can anyone help me with any thoughts.

Re: Migration of datastage code to Talend

Jesperrekuh — Mon, 27 Aug 2018 10:59:34 GMT

Not possible. Im not sure if I understand your question, what do you mean with "directly" ?
Side question, Im curious, why and for what reason?

Re: Migration of datastage code to Talend

abhi90 — Mon, 27 Aug 2018 11:46:27 GMT

Hi @pushpsourav,

It is not directly Possible. You have to code in Talend. Though Talend will add lots of features which can not be achived directly in DataStage like FileWatcher,archiving in s3.

Re: Migration of datastage code to Talend

Anonymous — Mon, 27 Aug 2018 14:58:17 GMT

what FileWatcher,archiving do ?

Re: Migration of datastage code to Talend

abhi90 — Mon, 27 Aug 2018 16:08:39 GMT

Like checking if any File comes then only trigger Staging Process--FileWatcher

To Send your File after your end to end Process to a Location(s3/Linux) after your Job completition-Archival

Re: Migration of datastage code to Talend

Anonymous — Tue, 28 Aug 2018 13:25:45 GMT

ohh that's indeed a good thing. For such things we were using shell scripts to do such tasks in Datastage

Re: Migration of datastage code to Talend

Anonymous — Tue, 04 Dec 2018 05:45:49 GMT

Hi pushpsourav,

Good to know that you are also doing ETL Jobs migration from DataStage to Talend. I'm sure you and your team might be half way on your journey for full migration.

I'm working for a company which hired me as a Business Analyst for ETL Jobs migration from DataStage to Talend. As this is my first BA job in IT, i want to know what are the best practices and where to kick start the project? Any help is greatly appreciated.

Below are my questions

1) Where should i start my analysis?

2) What all the documents i need to create?

3) Do you have any existing documentation. You can delete your client names and send it please.. i can give you my person email id, if you would like to.

Thanks a ton in advance.

Ramesh

Re: Migration of datastage code to Talend

Jesperrekuh — Tue, 04 Dec 2018 18:25:03 GMT

@rchinta

Im not migrating from Datastage... to ... talend... however, you're the business analyst and should know where to start to be honest.

However I do get your question on where to start regarding Talend and hope.

Many times I end up in discussions ... why using talend... or ... why not this or why not that.

My advice, for what its worth, start with 3 things.
> Support, monitoring, performance, scheduling.

>> This is highly underestimated. Where to store or write your log-files to.

>> Go for log4j and make sure its proper implemented in your etl-jobs

>> Make sure the teams fully understand tLogCatcher. tStatcatcher tFlowmeter. Use is wisely

>> Configuration of your jobs, think how to maintain configuration of your processes and the way its implemented...

>> When and how to process your jobs. Atomic approach,

> (Technical) Datasources Integration

>> If you have a lot of api / web / ftp go for cUrl dont even bother using Talend components, components work but eventual these REST and SOAP components wont work as you wanted they're limited (search it).

>> Files, Databases, no problems at all.

> Seperation of concerns.

>> Keep your config manageable outside of talend. in a custom json, xml (you have to design this yourself)

>> Extraction layer, processing layer, storage and re-running whenever your jobs failed.

>> Consider implementing middleware as an interface to your etl-processes. Often etl-teams try and create middleware processes to control this type of process. Take a look at node-js and its roll/function in your data (etl) landscape

>> Keep it simple.

No docs sorry.

Cheers and good luck

Jesper

Re: Migration of datastage code to Talend

Anonymous — Wed, 05 Dec 2018 04:17:16 GMT

Thank you very much for your reply. I really appreciate that.

Re: Migration of datastage code to Talend

Anonymous — Tue, 10 Sep 2019 06:45:18 GMT

Hello,

Do you suggest importing .dsx files into Talend to get the basic structure and skeleton of the Datastage Jobs.

While doing so, Talend is missing MITI and Datastage classes. I believe they are probably routines that need to be added. Can you tell me from where I can get them?

Thanks for the help!

Re: Migration of datastage code to Talend

Jesperrekuh — Wed, 11 Sep 2019 14:31:20 GMT

Yeah why not ... your goal is migration ( I assume ) ? or integration , like part of your daily ops.

IBM should have it available via api's / java packages ... you can easily add libraries to Talend via preferences. If you want to build custom components follow the tutorial to speed things up.

IBM Docs

But I would not recommend importing it (no experience in doing so) , it probably works to 'generate' your blueprint, but Talend is designed with its own "philosophy" and in a way you should stick with that. I would only import/export (metadata)schemas... to easy datamodelling part.

Seperate concerns and avoid getting all mixed up due to introducing different techniques, conversions and even versions. Before you know it your Talend proces relies on package X and the company upgraded to Y.

Re: Migration of datastage code to Talend

Anonymous — Thu, 12 Sep 2019 09:18:04 GMT

Hello,

Thanks for writing back.

I am trying to migrate my Datastage jobs to Java using Talend by importing the .dsx files and I completely agree with you that it is a mess to go by this process, but I have a huge number of jobs at hand and this is the fastest and most cost effective way for me. As Talend is able to give me something while importing a job I am not sure why it is not able to detect it.
For example, If conditions of Datastage are being converted to MITI.IFF but it is no where in the system routines hence it is not being detected. Is it related to this MIMB?

A DataStage function JulianDayFromDate is being converted to DataStage.JulianDayFromDate but is not getting resolved.

So the problem goes on for all the functions of Datastage so I just want to confirm if I am not missing out on importing any package/routine for the same.

I would also appreciate if there is anything that you/the community can suggest me for my problem.

Thanks