topic Re: List File names that are NOT present in HDFS Directory in Talend Studio

List File names that are NOT present in HDFS Directory

rohitpatil1993 — Fri, 18 Jun 2021 09:21:51 GMT

Hello

I am trying to get the names of Files that are NOT present in the specified HDFS directory.

(Note: I have multiple files using date mask for this check)

Here is what I thought will work but It's not

If condition --> !((Boolean)globalMap.get("tHDFSExist_1_EXISTS"))

tJava --> System.out.println(((String)globalMap.get("tHDFSExist_1_FILENAME")));

Can anyone help with the solution ?

Re: List File names that are NOT present in HDFS Directory

Anonymous — Fri, 18 Jun 2021 09:43:27 GMT

How do you set the file name parameter of tHDFSExist? use the global variable of tHDFSList? If so, all files do exist in the specified HDFS directory, and the condition you set is always false.

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Fri, 18 Jun 2021 11:30:20 GMT

I think the condition is right since I want the names of Files that are not present.

Here is what you asked for.

Re: List File names that are NOT present in HDFS Directory

Anonymous — Fri, 18 Jun 2021 13:25:50 GMT

You are iterating all files in the same directory as tHDFSList did, I don't understand why there will exist files are not present.

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Mon, 21 Jun 2021 05:43:53 GMT

Maybe I set the wrong logic...

What is the way to list the files not present in the specified directory?

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Tue, 22 Jun 2021 04:24:20 GMT

@Xiaodi Shi @Manohar B @Fred Trebuchet

Do you guys have any idea on how to achieve the desired result?

Re: List File names that are NOT present in HDFS Directory

manodwhb — Tue, 22 Jun 2021 05:54:10 GMT

@Rohit Patil ,what the location of the folder to check the List and exist components?

Thanks,

Manohar

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Tue, 22 Jun 2021 06:41:06 GMT

The location is set in tHDFSList.

Re: List File names that are NOT present in HDFS Directory

Anonymous — Tue, 22 Jun 2021 07:05:36 GMT

tHDFSList is used to iterate all files in the specified folder, the files must exist if you use tHDFSExist to check it. It's a logical error.

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Tue, 22 Jun 2021 07:13:30 GMT

Yes. I understood that it was a logical error but what's the way out?

@Shicong Hong

Re: List File names that are NOT present in HDFS Directory

Anonymous — Tue, 22 Jun 2021 07:21:35 GMT

Can you let me know how the name of files that you need to check if it exists or not on HDFS server come? Can you explain your requirements with more details?

Re: List File names that are NOT present in HDFS Directory

manodwhb — Tue, 22 Jun 2021 08:01:36 GMT

@Rohit Patil, Usually kind of this scenario want to check if one location files are present in other location then you can do the your job way. But here I do not understand why are you checking the file after listing in same directory. it will be always true.

Thanks,

Manohar

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Tue, 22 Jun 2021 08:18:08 GMT

Here is the requirement :

I have total 11 files which need to be checked if they are present in HDFS directory. If all are present then we go ahead with other steps & If one or more files is not present, then I have to send those missing file names via email.

Now file names look like this

"officeAddress_"+((String)globalMap.get("currentDate"))

"employeeAddress_"+((String)globalMap.get("currentDate"))

"xyz_"+((String)globalMap.get("previousDate"))

Note : currentDate and previousDate are global variable defined in start of job.

@Shicong Hong

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Tue, 22 Jun 2021 08:38:09 GMT

Please ignore the attached solution I added in question body...The logic is wrong :

Here is the requirement :

Now file names look like this

"officeAddress_"+((String)globalMap.get("currentDate"))

"employeeAddress_"+((String)globalMap.get("currentDate"))

"xyz_"+((String)globalMap.get("previousDate"))

Note : currentDate and previousDate are global variable defined in start of job.

@Manohar B

Re: List File names that are NOT present in HDFS Directory

Anonymous — Tue, 22 Jun 2021 16:34:04 GMT

Assuming the 11 file names are read from somewhere in the start of job, you need to iterate the rows and pass the current file name to tHDFSExist, for example:

tFileInputDelimited--main--tFlowToITerate--Iterate-->tHDFSXist--runIf---tJava

Re: List File names that are NOT present in HDFS Directory

rohitpatil1993 — Wed, 23 Jun 2021 05:50:08 GMT

@Shicong Hong I have developed the half part --> Which checks if all 11 files are present or not..Next part to find the names of missing files is the challenge.

Can you elaborate more

runIf---tJava ?

Re: List File names that are NOT present in HDFS Directory

Anonymous — Wed, 23 Jun 2021 09:21:11 GMT

Yes, as you did in the first post,

....runIf---tjava

on tJava: print the file name on the console for debugging.

If you want to output the file name to somewhere, use tFixedFlowInput to generate the file name, eg:

runIf--tFixedFlowInput--main--tFileOutputDelimited.