topic Re: Passing Context Variables in BigData Spark Job in Talend Studio

Passing Context Variables in BigData Spark Job

Anonymous — Sat, 16 Nov 2024 09:06:26 GMT

Hi Team

I have designed a Job to convert csv file to parquet format as shown in pic

In tfileinputdelimited i am using context variable to define file path

i am getting following error stating No input path specified

cant we parameterise BigDataSpark Job.

How this can be resolved. Do we have any other approch

please help me out.

ErrorMessage

par_dir_name...HR Services Action Report 02.07.2016.csv
[ERROR]: org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand - Aborting job.
java.io.IOException: No input paths specified in job
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:201)
at org.talend.hadoop.mapred.lib.file.TDelimitedFileInputFormat.listStatus(TDelimitedFileInputFormat.java:70)
at org.talend.hadoop.mapred.lib.file.TDelimitedFileInputFormat.getSplits(TDelimitedFileInputFormat.java:96)

Thanks

Re: Passing Context Variables in BigData Spark Job

Anonymous — Mon, 06 Nov 2017 07:18:50 GMT

Hello,

Can you successfully execute your spark job when use your file path directly in tfileinputdelimited component without context variables?

Best regards

Sabrina

Re: Passing Context Variables in BigData Spark Job

badri-nair — Tue, 12 Jun 2018 15:04:33 GMT

Hi Sabrina,

I have tried the same. Context load component does not work in a spark batch job.

the hard coded path to files and directories work.

Thanks

Badri Nair

Re: Passing Context Variables in BigData Spark Job

torradot — Fri, 09 Nov 2018 17:19:09 GMT

Hello

Have you some news about this topic ? I have the same problem to iterate several files with a variable name in the tfileinputdelimited .

Thanks for the help...

Tiago

Re: Passing Context Variables in BigData Spark Job

How1 — Mon, 28 Jan 2019 11:58:36 GMT

We have the exact same issue, I would welcome any feedback on this at all.

Hardcoded values in the tFileInputDelimited work fine, but the minute I want the filename to be a context variable, the job fails with a not found error.

Even if we have the tFileInputDelimited path as:

"C:\test\" + context.foldername + "\file.csv"

the Spark job fails with:

org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: C:\test\file.csv