First FTP Job - Copy files on FTP server - Split XML file per order into a new XML file - Upload new files with ActiveMQ

[Oliemar84]

Hello everybody,

I'm assigned to create a job, however I've never done anything with FTP nor XML components before. (only SQL and retreive/update DB)

The documentation can only show me as much as the general info, but I haven't seen anything that can help me further with where I'm stuck on.

My mentor is unfortunately unavailable and I hoped that someone could get me started here /steer me in the right direction.

I've broken down the job as per below what needs to be done.

The Job;

1. I would need to establish a connection with the FTP.
2. Check whether new files are available, if not, the job should end.
3. Copy all files (orders.xml and perhaps others) from FTP folder "\orders\" into "\orders\archive\"
4. if orders.xml contains more than 1 order, the xml should be split into seperate xml files based on the order.
5. The new xml-files/orders should be send with AcriveMQ Artemis with orders.xml to be renamed as header "OriginalFilename"

Up to now I have below schedule created;

 

1. FTPConnection is created.
2. I would assume, I would need tFTPFileProperties, in between FTPConnection and tFTPGet to check if the orders.xml was updated in "\orders\". But how can I do this? I assume I need some Java code for this? Or perhaps a different component?
3. FTPGet would get all files and store it locally. Can I put it directly into "\orders\archive\" with this component into the FTP server? Or should I use tFTPut as in the schedule to store it into the FTP?
4. I really have no clue how to split the xml files per order. Any help, links or examples would be greatly appreciated.
5. To be honest I have not yet looked into ActiveMQ, given I can't sort the above out yet, I figured I would wait to dig into ActiveMQ till I have the first 4 steps working.

Many thanks in advance for your help!

[Anonymous]

This is not the best way to ask a question here I am afraid. You have essentially asked for a complete solution which is made up of multiple problems. You will need to first solve the FTP part of this and then go to the XML processing part.

To start you off, if you want to find check whether the files exist, you will need to first use the tFTPFileList component. This will allow you to carry out a search of your FTP location for certain file patterns. It will return the records via an Iterate flow, so you'll probably want to use a tIterateToFlow component to work with this data.

Have a look at this example (https://help.talend.com/r/en-US/7.3/ftp/tftpfilelist-tftpconnection-tlogrow-listing-and-getting-files-folders-on-ftp-directory-standard-component) which shows this in a very simple way. Then try and extrapolate from this to get to solving the first part of your issue.