topic Re: retrieve if a file exists in an http server in Talend Studio

retrieve if a file exists in an http server

Anonymous — Mon, 25 Jan 2016 09:05:04 GMT

Hello,
we have a large list of file stored in a http server with a database storing the path and file name.
Is there a way to iterate for each file in the database and retrieve the http error code (ex. 404)?
Thanks,

Re: retrieve if a file exists in an http server

Anonymous — Thu, 28 Jan 2016 12:59:04 GMT

Hi
Read the file path or file name from database, and then iterate each one and fetch it from HTTP serve with tFileFetch, uncheck 'die on error' option if you want the job continue to retrieve next file even there is an error. eg:
tMysqlInput--main--tFlowToIterate--iterate--tFileFetch

Re: retrieve if a file exists in an http server

Anonymous — Thu, 28 Jan 2016 14:09:23 GMT

Thanks,
but I do not want to retrieve the file but to ensure the file is present, as we have many files which are not existing or has been moved to anohter location.
So it is to get the status when the file does not exist anymore at the stored location
cheers.

Re: retrieve if a file exists in an http server

Anonymous — Fri, 29 Jan 2016 04:16:49 GMT

Hi
Write a routine function to do it, here is an example code in this page:
http://stackoverflow.com/questions/4596447/check-if-file-exists-on-remote-server-using-its-url
Call the routine function on tMap or tJavaRow, eg:
tMysqlInput--main--tJavaRow--main--tLogRow

Re: retrieve if a file exists in an http server

Anonymous — Fri, 29 Jan 2016 07:44:20 GMT

Ok this is the first time I will use a routine.
I have create a new routine call HttpFileCheck.
And I paste the code from the link you have provided, is it correct like below?
Thanks,
package routines;
public class HttpFileCheck {
    import java.net.*;
    import java.io.*;
    public static boolean exists(String URLName){
        try {
          HttpURLConnection.setFollowRedirects(false);
          // note : you may also need
          //        HttpURLConnection.setInstanceFollowRedirects(false)
          HttpURLConnection con =
             (HttpURLConnection) new URL(URLName).openConnection();
          con.setRequestMethod("HEAD");
          return (con.getResponseCode() == HttpURLConnection.HTTP_OK);
        }
        catch (Exception e) {
           e.printStackTrace();
           return false;
        }
    }
}

Re: retrieve if a file exists in an http server

Anonymous — Sat, 30 Jan 2016 10:58:04 GMT

Hi
I did't test the code, but I think it is OK for you to check the file if it exists. Please test it and let me know if it is OK or any problem you get.