Talend Data Integration: Reading FTP files directly without downloading them to the local system

[Shicong_Hong]

The dedicated tFTPxxx components do not support reading files directly from FTP servers. Usually the file is obtained from the FTP server to the local system first and then the file is read. However, this can cause performance issues when many files or large files need to be read.

Resolution

One solution is to use the JSch library, which allows the creation of an InputStream object from a file via SFTP. Then, use the tFileInputDelimited component to read data from the InputStream object. This avoids downloading the file to the local system.

For example, in the following Job, the Java code iterates over the FTP files one by one, using a JSch library file and a tFileInputDelimited component to read the file directly.

The tLibraryLoad component loads the jsch-0.1.55.jar JSch library file, then the tJava component uses the following code to create the InputStream object for the current FTP file.

JSch jsch = new JSch();
        Session session = null;
        try {
            session = jsch.getSession("username", "localhost", 22);
            session.setConfig("StrictHostKeyChecking", "no");
            session.setPassword("password");
            session.connect();

            Channel channel = session.openChannel("sftp");
            channel.connect();
            ChannelSftp sftpChannel = (ChannelSftp) channel;
            java.io.InputStream in=sftpChannel.get( ((String)globalMap.get("tFTPFileList_1_CURRENT_FILE")));
            globalMap.put("sftp_inputStream",in);
        
        } catch (JSchException e) {
            e.printStackTrace();  
        } catch (SftpException e) {
            e.printStackTrace();
        }

The tFileInputDelimited component reads the data from the InputStream object:

Environment

Talend Data Integration