Skip to main content
Announcements
Qlik Introduces a New Era of Visualization! READ ALL ABOUT IT
cancel
Showing results for 
Search instead for 
Did you mean: 
Tech_wiz
Contributor
Contributor

Can we able to integrate continuous data loading with talend from SFTP?

In First yellow crossed mark i have set the variable with the value of file name and in second yellow crossed mark I have set the variable with the value of field selector which in between the input and output

So can we able to get the filename from SFTP location or any S3 lambda function and pass through this pipeline variable and also can we able to set the value of field selector to 2nd variable(output)

Labels (2)
2 Replies
Jean-François_Delhumeau

Hello,

For now, context variable can't be operationalized meaning that the value of variables is defined inside the pipeline. This feature is part of our roadmap for 2023.

Afoster1677824530
Contributor III
Contributor III

Hello,

Very nice information , I want to suggest you dear hope so it will helps you As you said above Yes, it is possible to integrate continuous data loading from an SFTP server with Talend.

 

Talend provides a number of components that can be used to connect to an SFTP server, retrieve data, and load it into a target system or database. Some of the key components that you may use for this purpose include tFTPGet, tSFTPFileList, and tSFTPGet.

 

To set up continuous data loading, you can use the Talend Job Scheduler to schedule the data retrieval and loading process to run on a recurring basis. You can set up the Job Scheduler to run at specific intervals, such as every hour or every day, to ensure that data is loaded continuously. Connect EBT Customer Service

 

 

 Additionally, you can use Talend's Change Data Capture (CDC) components to capture changes to data in real-time and load them into a target system. The CDC components can monitor specific tables or columns in a database and capture any changes that occur, allowing you to load only the changed data into the target system.

 

Overall, Talend provides a range of tools and components that can be used to integrate continuous data loading from an SFTP server. However, the specific configuration and usage of these components may vary depending on the requirements of your workflow and the data you are working with. please also give your feedback .