topic Re: File Transfer in Talend Studio

File Transfer

ar7368 — Fri, 23 Mar 2012 01:02:21 GMT

Hi,
I have a job to modify filenames of files(pdf) from source with static data inserted in the fourth component of file name such that it will have the following format
TAC_CAT_time/date_Op2_uniqueID.pdf. The filenames will already contain TAC_CAT_DATE_unique ID.pdf. thus Op2 (case sensitive) is missing. The TAC and CAT components are 8 and 9 characters long respectively. I have figured out all of the above, except that when the MRN or HAR are not within 8 or 9 characters long i need to generate a error. Sometimes these components are keyed in wrong by users. How do I filter these files? to generate a email.
So in effect the 28th character in my filename needs to "_".

Re: File Transfer

Anonymous — Fri, 23 Mar 2012 05:57:34 GMT

So in effect the 28th character in my filename needs to "_".

According to your requirement, we can create part of your job as follow.
Regards,
Pedro

Re: File Transfer

ar7368 — Fri, 23 Mar 2012 13:03:43 GMT

Hi Pedro,
Sorry its the 27th character, if you start from 0. So right now I have the following code in my tfilecopy component
---------------
((String)globalMap.get("tForeach_1_CURRENT_VALUE")).substring(0,19)+((String)globalMap.get("tForeach_1_CURRENT_VALUE")).substring(23,27)+((String)globalMap.get("tForeach_1_CURRENT_VALUE")).substring(19,23)+"_"+"Op2"+"_"+((String)globalMap.get("tForeach_1_CURRENT_VALUE")).substring(28,((String)globalMap.get("tForeach_1_CURRENT_VALUE")).length())
----------------
So should I be using tIterateflow instead of tforeach?. Will try as you suggested.
Thanks,

Re: File Transfer

ar7368 — Fri, 23 Mar 2012 16:33:18 GMT

Hi Pedro,
The job ran as you suggested. But in the destination folder I can see just one file that is transferred. The rest of the files are missing eventhough the job says it moved 11 files which is the correct number. I'm missing something here. Please help.
Thanks,

Re: File Transfer

Anonymous — Mon, 26 Mar 2012 03:13:42 GMT

Hi
From this image, we are sure that tFilerRow works fine.
It seems that there is something wrong in tFileCopy.
You'd better recheck it.
Regards,
Pedro

Re: File Transfer

ar7368 — Mon, 26 Mar 2012 14:38:27 GMT

Hi Pedro,
I just can't isolate the problem, Following are the screens when I use tlogrow and alternatively when I use tfilecopy
The destination filename code I have on Tfilecopy is as follows
----------------------------
((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(0,19)+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(23,27)+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(19,23)+"_"+"Op2"+"_"+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(32,((String)globalMap.get("tFileList_1_CURRENT_FILE")).length())
-------------------------------
When Tfilecopy is used with Tfilter will the files transfer or is Tfilter used to just list the filtered files but no physically move them?

Re: File Transfer

ar7368 — Mon, 26 Mar 2012 15:17:32 GMT

Sorry that is not the case. Its 5 parts before the insertion of "Op2" a constant. The destination files need to have 6 parts in total.

Re: File Transfer

janhess — Mon, 26 Mar 2012 15:29:12 GMT

I've got this routine for splitting strings into parts on a delimiter.

    /**
     * reverseIt() Reverses a string.
     * 
     * {talendTypes} String
     * 
     * {Category} Ncl_routines
     * 
     * {param} string ("fred") input: the string to be reversed
     * 
     * {example} reverseIt("fred") # derf
     * 
     */  
	public static String reverseIt(String source) {
		    int i, len = source.length();
		    StringBuffer dest = new StringBuffer(len);
		    for (i = (len - 1); i >= 0; i--)
		      dest.append(source.charAt(i));
		    return dest.toString();
	}
   /**
     * anyWord() finds a word in a string based on a delimiter and the number of the word required.
     * It can search forwards or backwards depending on whether the number of the word is positive or negative.
     * 
     * {talendTypes} String
     * 
     * {Category} Ncl_routines
     * 
     * {param} string("hello Example") input: The string need to be parsed.
     * 
     * {param} string(" ") input: the delimiter
     * 
     * {param} int(0) input: the number of the word to return - 0 is first occurrence -1 is the last.
     * 
     * {example} anyWord("hello Example"," ",0) # hello
     */
 
	@SuppressWarnings("finally")
	public static String anyWord(String message, String delimiter, Integer number ) {
    	String returnString = null;
    	Integer reverseit = 0;
    	try {
	    	if (number < 0) {
	    		message = Ncl_routines.reverseIt(message);
	    		number = -1 - number;
	    		reverseit = 1;
	    }
    	if (delimiter.equals(" ")) {
    		String[] starwords = message.split (" ");
    		returnString = starwords;
    	}
    	if (! delimiter.equals(" ")) {
    		String[] starwords = message.split ("\\" + delimiter);
    		returnString = starwords;
    	}
    	if (reverseit == 1) {
    		returnString = Ncl_routines.reverseIt(returnString);
    	}
    	} catch (ArrayIndexOutOfBoundsException e) {
    		returnString = "";
    	} finally {
    		return returnString;
    	}
    } 
}

So for your string you could have
anyWord(((String)globalMap.get("tFileList_1_CURRENT_FILE")),"-",0) + "_" + anyWord(((String)globalMap.get("tFileList_1_CURRENT_FILE")),"-",1) + "_" +
anyWord(((String)globalMap.get("tFileList_1_CURRENT_FILE")),"-",2) + "_" +
anyWord(((String)globalMap.get("tFileList_1_CURRENT_FILE")),"-",3) + "_" +
"Op2_" +
anyWord(((String)globalMap.get("tFileList_1_CURRENT_FILE")),"-",4)

This saves worying about the length of each component.
"

Re: File Transfer

ar7368 — Mon, 26 Mar 2012 15:42:30 GMT

Ok the third component is a date which is in mmddyyyy, I need it in yyyymmdd. So how do i deal with that?. I am currently using the following in the 3rd component.
-----------
((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(23,27)+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(19,23)
-----------
Could I use the one above in the 3rd part of your code ?

Re: File Transfer

janhess — Mon, 26 Mar 2012 16:55:25 GMT

Do the substring on the anyWord.

Re: File Transfer

ar7368 — Tue, 27 Mar 2012 15:35:30 GMT

The problems still persists. There is just one file each ( one from the filter and other from reject) moved to the destination folders. Why is this?. I have no clue why just one file is moved to the destination although the job says otherwise.

Re: File Transfer

janhess — Tue, 27 Mar 2012 16:02:48 GMT

What are the input and output file names for your file copy?
Your tFilterRow should use .equals not ==

Re: File Transfer

ar7368 — Tue, 27 Mar 2012 16:07:56 GMT

For tfilecopy1:
Filename is : ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
Destiantion Filename : ((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(0,19)+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(23,27)+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(19,23)+"_"+"Op2"+"_"+((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(32,((String)globalMap.get("tFileList_1_CURRENT_FILE")).length())
--------------------------------------------------------------------------
For tfilecopy 3: Filename is : ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
Destiantion : Local drive.

I know you suggested using routines but I haven't. Still there is no explanation to why just a single file gets moved but not the rest

Re: File Transfer

janhess — Tue, 27 Mar 2012 16:09:41 GMT

You need to use the file name from tFilterRow for the output file name. The one from tFileList will always be the same as it's before the iterate.

Re: File Transfer

ar7368 — Tue, 27 Mar 2012 16:22:28 GMT

you mean in Rename - destination fielname?

Re: File Transfer

janhess — Tue, 27 Mar 2012 16:46:10 GMT

Sorry I'm talking rubbish - tFileList file name is after the iteration.
What does the tLogRow show for the file name in your earlier example?

Re: File Transfer

ar7368 — Tue, 27 Mar 2012 16:53:02 GMT

Ok, the tlogrows shows the rejects and filtered files according to the condition specified in tfilterrows. I have merely replace the tlogrows with tfilecopy in the earlier example.

Re: File Transfer

ar7368 — Tue, 27 Mar 2012 16:58:03 GMT

When I hit ctrl+space in Tfilecopy destination rename should I be choosing
"((Integer)globalMap.get("tFilterRow_3_NB_LINE_OK"))"... but what is the Integer part?.

Re: File Transfer

janhess — Tue, 27 Mar 2012 17:07:35 GMT

OK I think the input file name is in the column name that comes out of tFilterRow - that's why it's displayed in tLogRow.
Does the file that's being created look like any of the names you're expecting?

Re: File Transfer

ar7368 — Tue, 27 Mar 2012 17:12:44 GMT

yes in the tfilecopy( good files) i see one file with the modifications I have specified. Similarly in tfilecopy(reject files), I see one file that is not according to specifications. So of the 20 rows processed 9 are bad and 11 good. Problem: I see just one of each category in their respective destinations.