topic Re: Big Data Spark Job - Process job based on file name in Talend Studio

Big Data Spark Job - Process job based on file name

Anonymous — Sat, 16 Nov 2024 09:47:59 GMT

Hi,

I am creating big data spark job and have to check file name before proceeding to the specif workflow like

checkfilename -> (if contains "abc") -> then read file / process file

(if filename contains "xyz") -> then read file/ diff process

tInputfileDelimited -> Checkfilename (if contains "abc") -> then process file

(if filename contains "xyz") -> then diff process

Which component I have to use to implement above scenario. I tried to use tJavarow but runif has several limitation. We can connect tjavarow ->(runif)-> tJava only.

Re: Big Data Spark Job - Process job based on file name

Anonymous — Mon, 15 May 2017 10:47:06 GMT

Hi,

In a standard job, here is a component tfileproperties which can create a single row flow that displays the properties of the processed file.

Best regards

Sabrina

Re: Big Data Spark Job - Process job based on file name

mamohan — Mon, 08 Apr 2019 21:09:17 GMT

I have similar requirement and tfileproperties is not available in spark job. Can you please suggest for a spark job.