Moving to the cloud methods: Lift and Shift, Refactor, and Rearchitect

[TalendSolutionExpert]

Historically born for on-premises systems, Talend started as an open-source data integration company with unique data quality features and quickly grew the product to make big data integration easy for customers. Talend Cloud on AWS, the first iteration of a cloud data integration platform, was introduced in 2015. Since then, Talend has expanded the cloud value, providing a multi-cloud integration platform with unique features.

Now Talend Cloud platform, with hybrid deployment capabilities, is mature enough to cover all cloud, as well as historical on-premises, use cases. By moving to Talend in the cloud, customers can do even more using cloud-only services and capabilities, such as Talend managed Execution Engines, Pipeline Designer, Data Inventory with the Talend Trust Score, API Services, and Stitch Data Loader.

For more information on the benefits of Talend Cloud platform, see Moving to cloud? What are the benefits of Talend Data Fabric in the cloud.

There are multiple paths to move to the cloud and the path you choose depends on two main factors:

• Skill set that you have in the company

• Modularity within your existing workloads

Moving to Talend in the cloud, choose your strategy

In general, when moving to the cloud, organizations follow one of three main strategies: Lift and shift, re-architect and refactor. Each of these strategies require different considerations when considering a move to Talend in the cloud.

Lift and shift

As the name implies, this method involves lifting your entire stack and shifting it from an on-premises environment to hosting it in the cloud. In other words, you will move a copy of your current environment, without making extensive changes, to the cloud. Once this is done, you no longer have to manage or maintain on-premises hardware for your Talend deployment.

Source code management

Talend in the cloud supports Git based source control only. If you are currently using SVN or other non-Git based source code management, you’ll need to migrate. You can find tutorials on Qlik Community to help with this process.

By moving to Git based source code management you’ll be able to use source code management (SCM) services. A key advantage of this is that hosted SCM services usually have DevOps capabilities that can be easily integrated with the continuous integration capabilities of Talend Cloud platform.

Development tools

Talend Studio is your unified development environment whether you are using Talend on-premises or Talend in the cloud. Moving to Talend Cloud platform gives you the opportunity to use an innovative set of development tools such as: Pipeline Designer, Stitch Data Loader, and API Services (API Designer and API Tester).

Execution engines

If you are using JobServers in your current on-premises environment, Talend Remote Engine is the recommended execution engine for Talend in the cloud. Like JobServers, remote engines are installed and managed by you, have unlimited resources, and are ideal for hybrid architecture deployments.

If your data sources or targets are behind a firewall or if the data that needs to be processed by Talend Jobs must stay in its original location – for example due to data jurisdiction, security or compliance requirements – remote engines are ideal. This is because remote engines only send status and execution logs to Talend Management Console with no data being transited through Talend Cloud platform.

Admin assets, projects, and tasks

To support your move to the cloud, Talend has developed a number of tools to facilitate your migration, which are included in Talend Cloud Migration Platform (TCMP). TCMP, is available in the download section of Talend in the cloud or from the Talend Community Move to Cloud > Cloud Migration Tools page. It moves your on-premise Users, Groups, Projects, Tasks, and Execution plans to Talend Cloud platform. With TCMP you’ll be able to easily migrate admin assets from multiple Talend Administration Console environments and different versions simultaneously from a single application.

Replatform

As a variation of lift and shift, replatforming involves making a few adjustments to your existing environment to optimize your landscape for the cloud. This allows workloads to take advantage of cloud functionality and cloud cost optimization without the high level of resources you had on-premises. The core architecture of applications stays the same, while allowing you to take advantage of these cloud capabilities for better scaling and resources management.

Managed execution engines

A managed execution engine or Talend Cloud Engine is a compute resource managed by Talend in the cloud. It runs on demand to executes Job tasks. It has fixed physical specification (2 vCPU, 8 GB Ram, 200 GB), can run up to three tasks in parallel, and requires sources and targets to be visible through the internet.

Talend Cloud Engines are completely preconfigured, making it easy to start right away in the cloud with just a couple clicks in Talend Management Console. If you move/synchronize data from SaaS applications or cloud platforms and if the cloud engines' fixed resources are sufficient for your workloads, this could be the right choice for you.

Infrastructure as Code

Infrastructure as Code (IaC) is the process of managing and provisioning resources (compute, storage, and others) through code rather than through physical hardware configuration or interactive configuration tools. IaC can greatly simplify and accelerate your infrastructure provisioning processes, help you avoid mistakes and comply with policies, keep your environments consistent, and save your company a lot of time and money.

IaC should be a crucial aspect of your move to the cloud. If you think of cloud computing as the first step to solving many of the problems caused by manual IT management, then it’s fair to say the IaC is the next logical step. It takes cloud computing to its fullest potential, and it frees developers and other professionals from performing manual, error-prone tasks.

Talend Remote Engines being available on Azure and AWS marketplaces allows you to programmatically scale, provision and deploy, and start and stop your Remotes Engines. The Talend Management Console API has Remote Engine management automation methods in order to create, delete and pair Remote Engines programmatically (TMC API). This allows you with unfettered elasticity – the ability to easily scale up or down your usage of Talend in the cloud and the needed Azure or AWS resources.

DevOps culture and Zero Install Continuous Integration (CI)

If your organization does not currently ascribe to the Software Development Life Cycle methodology, moving to the cloud is the right time to make the move and adopt the DevOps culture. Talend is compliant with Java and Maven standards, meaning that you can use a classic CI server application such as Jenkins and a common artifact repository like Nexus or Artifactory. In the cloud, a Talend Job can be built, delivered and deployed in two different ways: you can either build an Artifact, deliver it to a Talend Management Console repository, and deploy it to a Remote Engine, Cloud Engine, or build a Docker image to be delivered on a Docker Registry and deployed to container services.

When implementing CI/CD, Talend recommends dedicating a CI secondary machine that contains prerequisites to build a Talend Job, such as Talend CommandLine and Maven plugins (CI builder and Cloud Publisher). 

Refactor

The Refactor approach is driven by a strong desire to improve your environment and represents the opposite of a lift-and-shift migration. Under this approach, it is assumed that the business needs to leverage cloud capabilities that are not available in its existing environment. In some instances, this requires re-engineering the application logic completely and developing cloud-native versions from scratch. In the long run, refactoring can be more cost-efficient because of the additional features added to particular applications.

Adopt cloud-only applications

The main value pillars for Talend Data Fabric in the cloud are innovation, lower total cost of ownership, scale, and security. Innovation is always a step ahead on Talend Cloud platform with monthly releases of new features, automatic upgrades, and most importantly, cloud-only capabilities and services. To illustrate this, take a look the Talend Willamette Dental Group case study.

Stitch Data Loader makes loading data to your cloud data warehouses and data lakes simple, secure and frictionless:

• Easily load data into a cloud data warehouse or data lake
• A managed data integration service you can trust
• Frictionless service that can scale and grow with Talend in the cloud

Pipeline Designer is the next generation self-service cloud data integration application:

• Schema on read, live preview, deploy anywhere
• Integrate structured or unstructured data in either batch or stream
• Addresses new users (data engineers, data scientists, analysts)

Talend API Services provides you with aa comprehensive API design and delivery platform.

• Enable collaborative API-first development
• Implement with Talend Studio
• Operate in Talend Management Console and integrate in your CI pipeline

Data Inventory with the Talend Trust Score maintains an inventory of data assets managed within Talend Cloud platform:

• Delivers a single point of data intelligence​ of data assets managed within Talend in the cloud
• Cuts the times it takes to find, understand and consume relevant datasets
• Access the Talend Trust Score to instantly assess the health and reliability of your data and it’s available for you to try here: https://www.talend.com/lp/trustscore/

Leverage new personas in your company

Talend cloud-only capabilities and services will help more non-technical persons to use data to bring value to your organization. New data user personas are emerging in most organizations, with many being subject matter experts with little or no technical background. Talend Cloud platform brings the right set of tools for them.

Data owners can govern their data in the cloud using Data Inventory, leveraging the Talend Trust Score to ensure that the data is of high quality and relevant. Citizen integrators no longer have to go to IT to run a new report. They can easily create a data integration or a data quality pipeline using Pipeline Designer on a web browser, even using their tablet. Data stewards can run a data quality campaign using Data Stewardship and business analyst can leverage Talend Data Preparation in the cloud to easily standardize data from multiple sources to help them make crucial decisions. All of these capabilities are available in a governed self-service fashion on a single cloud platform.

Conclusion

Which one is the best approach?

There is no absolute answer to this question, mainly because different use cases require different things. Picking one of the three approaches is a matter of finding the one that best suits your specific needs. Your organization's needs may vary and delve you into adopting a combination of these strategies