Workspace Management in Talend Cloud

[TalendSolutionExpert]

This article explained how the key concepts in managing Talend Cloud environments work together to allow you to effectively and safely create and deploy Jobs to Talend Cloud.

Content:

• Scenario
• Talend Cloud Environment
• Workspaces
• Talend Cloud artifacts
• Talend Cloud tasks
• Talend Cloud plans
• Talend Cloud project collaborators
• Talend Cloud workspace permissions
• Reviewing workspace management

Talend Cloud provides elements to centrally define, manage, orchestrate, execute, and promote artifacts published from Talend Studio or Talend Cloud Pipeline Designer. This article spotlights each of these elements to bring out their purpose, features, and relationship with other building blocks.

A better understanding of Talend Cloud workspace management concepts, including environments and runtimes, leads to better planned and organized workspaces and improved productivity for teams that use Talend Cloud.

Scenario

In this scenario, you build a project using Talend Cloud based solutions. From an SDLC perspective, some of your typical requirements are:

• Job development: Create Talend Jobs for one or more departments. Publish these Jobs in various environments, for example, DEV, SIT, UAT, PROD environments. You can also create your own separate environment based on your specific requirements. These environments would need to have a separate workspace for segregation based on, let's say, departments.

• Job scheduling: There is a need to schedule Talend Jobs and options to orchestrate the execution of more than one Job. Available execution engines need to be allocated to the environments based on their workload.

• Job permissions: Various teams working on the project, including development, testing, operations, and administrators, should be assigned to have permissions and roles to the environments and workspaces, to carry out their project-related activities.

The setup above can be easily built on Talend Cloud by leveraging various elements or building blocks offered at the platform level. First things first, let's talk about the foundation, which is Environments and Workspaces.

Talend Cloud Environment

A Talend Cloud environment is a self-contained space with the required resources (connections, engines, and so on) that allows the execution of tasks and plans in isolation, with no impacts on tasks and plans running in other environments.

For this scenario, you might decide to create Development (DEV), System Integration Testing (SIT), User Acceptance Testing (UAT), and Production (PROD) environments in Talend Cloud. If needed, you could also create other environments based on specific self-contained space and resource requirements.

Tips

• If you plan on using Webhooks for scheduling executions, ensure that the environment name does not contain any special characters, only alphanumeric characters and underscores.

• Allocate Cloud Engines to the environment proportionally to the number of concurrent executions you plan to run. Three different tasks or plans can be run in parallel on a Cloud Engine.

Workspaces

Talend Cloud workspaces are created inside an environment and allow a group of users to work together, preparing tasks and plans for execution. There is no limit to the number of workspaces you can create in an environment. However, environments are limited to the set of resources they share, so if you are adding more workspaces, ensure that you have the capacity to execute the tasks and plans that they will be used to create.

The most common arrangement for workspaces is that different teams or departments have their own workspace to create their own tasks and plans. While they will still share execution resources with other teams in the same environment, the developed artifacts can be managed in isolation from other teams.

In this scenario, a development team working with the HR department might create a Talend Job for fetching Workday data. They then publish the Job into a workspace owned by the HR team, in the Development (SIT) environment, and test it before promoting it to the Production (PROD) environment.

Tips

• Talend recommends using the same name for workspaces across environments. This is also a prerequisite for the promotion features to work.

• A user's role has no impact on a workspace.

• Groups cannot be added to a workspace.

• Each Talend Cloud account has a native Personal and Shared workspace.

• By default, you can use Talend Remote Engines in all the workspaces of an environment. You can restrict the use of an engine by allocating it to a specific workspace.

Talend Cloud artifacts

Having created environments and workspaces, you can publish Talend Cloud artifacts to a workspace. An artifact is a packaged Talend Studio Job that is published to Talend Cloud and contains (among other things) a JAR file, execution scripts, and log4j settings.

Artifacts are stored in a Talend Cloud artifact repository and are available from the workspace where they are published.

Tips

• Consider aligning the Git tag version to the artifact version

  • Git tag: v1.3.1-beta, publish with version: 1.3.1

  • Git tag: v2.0, publish with version: 2.0.0

Talend Cloud tasks

After building and publishing an artifact to a workspace, it must be executed with a specific configuration, such as setting the values of parameter values, specific database connections, and whether to use a Cloud or Remote Engine for execution.

Talend Cloud tasks (also referred to as Job tasks) combine artifacts published from Talend Studio with a specific execution configuration. When you have built your Job tasks, you can schedule your operations.

Tips

• A task can use an artifact from another workspace in the environment.

• Tasks can be cloned using the Copy feature.

• You can select Always use the latest available artifact version if you want your task to be updated automatically each time a new version of the artifact is published.

• Name tasks with the artifact name so they can be automatically updated upon publishing.

• Leverage custom connections for parameters that share the same values across tasks (allows management directly in Talend Cloud while reducing maintenance of tasks).

• Webhooks can trigger the execution of a Job task based on calls from external applications.

In this scenario, the artifact published to Talend Cloud workspace can be operationalized or executed after creating a task for it. Sometimes you need to run more than one task in an order or based on the results of another task execution. This is where a plan comes into the picture.

Talend Cloud plans

A Talend Cloud plan is a scheduled set of tasks from a workspace, organized into a combination of sequential and parallel steps for orchestration. Plans can consist of Job or pipeline tasks, or a mixture of them, depending on your license and requirements. Plans are created in a workspace but may reference tasks from other workspaces in the same environment. A Run Type execution trigger must be specified for a plan that could be Manual, Scheduled, or Webhook.

Tips

• Tasks can be included from multiple workspaces in the environment.

• Plans cannot be cloned.

• Only Job or pipeline tasks can be used in plans. Routes and data services are not supported.

• A plan cannot call another plan.

• For efficiency, consider using plans as scope for the Promotion feature.

• The same plan can only be in the execution queue (in Running status) a maximum of 50 times in a 60-minute time period.

• Webhooks can trigger the execution of a plan based on calls from external applications.

• Plans containing pipelines can only be created in the default environment.

• For plan, the failure handler can be a task or another plan.

For this scenario, as an example, you could consider creating a plan to run three dependent ETL Jobs in a sequence one after another, with the condition of successful execution of the previous Job. You could also configure the plan to run another handler data restore Job in case any of the ETL Jobs fail.

Talend Cloud project collaborators

Multiple users and groups may be assigned to work on a Talend Cloud project. Working with projects allows developers to collaborate on the same artifacts in the source repository.

Tips

• Leverage groups where possible to manage Talend project members.

• A user must have the Integration Developer role and must be a project collaborator if they plan to develop using Talend Studio.

• Any user can be a project collaborator, and this does not impact roles.

• To allow access to a Talend Studio project when using Git on-premises, the repository URL must end with a .git extension.

Going back to the scenario example, for the developer to check out the Talend project from their Studio, you need to provide them with project collaborator access on the Talend Cloud project, as well as direct access to the Git project.

Talend Cloud workspace permissions

Users can be granted view, publish, manage, author, or execute permissions on a Talend Cloud workspace. The following matrix illustrates the various workspace permissions in context to Talend Cloud artifacts, connections, resources, tasks, and plans (an X indicates that a workspace privilege gives access to a specific action).

		Artifact	Connection	Resource	Task		Plan
					Configuration	Execution	Configuration	Execution
Workspace Permissions	Publish	X
	Manage		X	X
	Author				X		X
	Execute					X		X

Tips

• Workspace permissions are set independently of roles and project collaborators.

• To publish an artifact to Talend Cloud, the user must have the Integration Developer role and be a project collaborator for the target project.

• Consider using shared workspaces organized around departments, lines of business, solutions, and so on in place of the default personal workspaces.

• Roles have no impact on the workspace and vice versa.

• Project collaborators have no impact on the workspace and vice versa.

In this scenario, you would provide the operations team members with manage, author, execute, and view permissions on a given workspace for them to be able to configure, execute, and monitor the connections, resources, plans, and tasks.

Reviewing workspace management

To demonstrate various configuration options and flexibility offered by Talend Cloud platform, explore the steps involved to achieve the following environment setup.

In the above environment setup, environment DEV has two workspaces: ABC and XYZ.

The following are the sequence of operations you would do on workspace ABC. Here the focus is to showcase task configuration.

1. Publish a Talend Studio Job as an artifact to Talend Cloud workspace ABC.

2. Create a task called Task1 from the artifact you uploaded. You can also specify the Job parameters, connections, resources, runtime, and run type in this step.

The following are the sequence of operations to be done on workspace XYZ. Here the focus is to showcase the plan configuration.

1. Create two tasks, Task2 and Task3, in workspace XYZ from the artifact published to workspace XYZ or ABC; either is possible.

2. Create a plan in workspace XYZ and add tasks Task2 and Task3. Add Task1, which belongs to workspace ABC, to the plan in workspace XYZ. Specify the Run Type for the plan.