Big Data Spark Job - Subjob and Logs creation

[Anonymous]

Hi,

I am creating big data spark job. I want to create one sub job and want to use into another job. Like standard job, we can use tBufferOutput. Which component we can use to create sub job?

I want to maintain the log as well for spark big data job. I am using tWarn -> tLogCatcher -> tLogRow -> tFileOutputDelimited in standard job. Which component I can use for big data spark job?

Thanks.

[Anonymous]

Thanks.... How I can save log4j output in a file with big data spark job?

[Anonymous]

Hi,

Please take a look at a custom tRedirectOutput component, which redirects all the console output to a file.

https://exchange.talend.com/#marketplaceproductoverview:marketplace=marketplace%252F1&p=marketplace%...

Best regards

Sabrina

[Anonymous]

Does this tRedirectOutput components works in Big Data Spark Job also....?

Basically my requirement is to create log of the spark job in a file which is generated at run-time in the spark job itself.

Can this be achieved through log4j....?

Any help will be much appreciated.

Thanks,

OJ

[Anonymous]

Hello,

This tRedirectOutput custom component will not available in talend bigdata spark job. Here is a KB article about https://community.talend.com/t5/Migration-Configuration-and/Cannot-Get-Talend-Spark-Job-Execution-Lo...

Feel free to let us know if it is OK with you.

Best regards

Sabrina