Migration of datastage code to Talend

[Anonymous]

Hi,

My team is planning to move on to Talend from Datastage. We are in initial stage right now, were just trying to figure out if we can migrate Datastage code directly to Talend.

Can anyone help me with any thoughts.

[Jesperrekuh]

Not possible. Im not sure if I understand your question, what do you mean with "directly" ?
Side question, Im curious, why and for what reason?

[abhi90]

Hi @pushpsourav,

It is not directly Possible. You have to code in Talend. Though Talend will add lots of features which can not be achived directly in DataStage like FileWatcher,archiving in s3.

[Anonymous]

what FileWatcher,archiving do ?

[abhi90]

Like checking if any File comes then only trigger Staging Process--FileWatcher

To Send your File after your end to end Process to a Location(s3/Linux) after your Job completition-Archival

[Anonymous]

ohh that's indeed a good thing. For such things we were using shell scripts to do such tasks in Datastage

[Anonymous]

Hi pushpsourav,

Good to know that you are also doing ETL Jobs migration from DataStage to Talend. I'm sure you and your team might be half way on your journey for full migration.

I'm working for a company which hired me as a Business Analyst for ETL Jobs migration from DataStage to Talend. As this is my first BA job in IT, i want to know what are the best practices and where to kick start the project? Any help is greatly appreciated.

Below are my questions 

1) Where should i start my analysis?

2) What all the documents i need to create?

3) Do you have any existing documentation. You can delete your client names and send it please.. i can give you my person email id, if you would like to.

Thanks a ton in advance.

Ramesh

[Jesperrekuh]

@rchinta

Im not migrating from Datastage... to ... talend... however, you're the business analyst and should know where to start to be honest.

However I do get your question on where to start regarding Talend and hope.

Many times I end up in discussions ... why using talend... or ... why not this or why not that. 

My advice, for what its worth, start with 3 things.
> Support, monitoring, performance, scheduling.

>> This is highly underestimated. Where to store or write your log-files to. 

>> Go for log4j and make sure its proper implemented in your etl-jobs

>> Make sure the teams fully understand tLogCatcher. tStatcatcher tFlowmeter. Use is wisely

>> Configuration of your jobs, think how to maintain configuration of your processes and the way its implemented...

>> When and how to process your jobs. Atomic approach, 

> (Technical) Datasources Integration

>> If you have a lot of api / web / ftp go for cUrl dont even bother using Talend components, components work but eventual these REST and SOAP components wont work as you wanted they're limited (search it).

>> Files, Databases, no problems at all.

> Seperation of concerns.

>> Keep your config manageable outside of talend. in a custom json, xml  (you have to design this yourself)

>> Extraction layer, processing layer, storage and re-running whenever your jobs failed.

>> Consider implementing middleware as an interface to your etl-processes. Often etl-teams try and create middleware processes to control this type of process.  Take a look at node-js and its roll/function in your data (etl) landscape

>> Keep it simple. 

No docs sorry.

Cheers and good luck

Jesper

[Anonymous]

Thank you very much for your reply. I really appreciate that.

[Anonymous]

Hello,

Do you suggest importing .dsx files into Talend to get the basic structure and skeleton of the Datastage Jobs.

While doing so, Talend is missing MITI and Datastage classes. I believe they are probably routines that need to be added. Can you tell me from where I can get them?

Thanks for the help!