https://community.qlik.com/t5/Official-Support-Articles/How-to-enable-server-side-encryption-in-S3-when-using-Spark/ta-p/2151046 <DIV class="talend-tkb-migrated-content"><DIV class="lia-message-template-question-zone"> <H2>Question</H2> <P>I want to use Spark Local with Amazon S3, and need to have server side encryption enabled in S3 when creating the target file. When using EMR or <STRONG>tHDFSConfiguration</STRONG>, I can set this property in the <STRONG>tHDFSConfig</STRONG> connector using <STRONG>fs.s3n.server-side-encryption-algorithm</STRONG>, but when using Spark Local there is no <STRONG>tHDFSConfiguration</STRONG>. How can I set this property?</P> <P>&nbsp;</P> <P>I've tried writing a local file and then using <STRONG>tS3Put</STRONG>, but this throws a compile error as there seems to be a jar collision when a Spark Local Job and a standard Job with <STRONG>tS3*</STRONG> are coupled together.</P> </DIV> <P>&nbsp;</P> <DIV class="lia-message-template-answer-zone"> <H2>Answer</H2> <P>You can set Hadoop configuration properties in the Spark Advanced Properties by adding the <STRONG>spark.hadoop.</STRONG> prefix. In this case, it should be <STRONG>spark.hadoop.fs.s3n.server-side-encryption-algorithm</STRONG>, and it then should be automatically injected into any Hadoop configurations that the Spark local Job creates.</P> </DIV></DIV> Tue, 06 Feb 2024 15:10:57 GMT TalendSolutionExpert 2024-02-06T15:10:57Z https://community.qlik.com/t5/Official-Support-Articles/How-to-enable-server-side-encryption-in-S3-when-using-Spark/ta-p/2151046 <DIV class="talend-tkb-migrated-content"><DIV class="lia-message-template-question-zone"> <H2>Question</H2> <P>I want to use Spark Local with Amazon S3, and need to have server side encryption enabled in S3 when creating the target file. When using EMR or <STRONG>tHDFSConfiguration</STRONG>, I can set this property in the <STRONG>tHDFSConfig</STRONG> connector using <STRONG>fs.s3n.server-side-encryption-algorithm</STRONG>, but when using Spark Local there is no <STRONG>tHDFSConfiguration</STRONG>. How can I set this property?</P> <P>&nbsp;</P> <P>I've tried writing a local file and then using <STRONG>tS3Put</STRONG>, but this throws a compile error as there seems to be a jar collision when a Spark Local Job and a standard Job with <STRONG>tS3*</STRONG> are coupled together.</P> </DIV> <P>&nbsp;</P> <DIV class="lia-message-template-answer-zone"> <H2>Answer</H2> <P>You can set Hadoop configuration properties in the Spark Advanced Properties by adding the <STRONG>spark.hadoop.</STRONG> prefix. In this case, it should be <STRONG>spark.hadoop.fs.s3n.server-side-encryption-algorithm</STRONG>, and it then should be automatically injected into any Hadoop configurations that the Spark local Job creates.</P> </DIV></DIV> Tue, 06 Feb 2024 15:10:57 GMT https://community.qlik.com/t5/Official-Support-Articles/How-to-enable-server-side-encryption-in-S3-when-using-Spark/ta-p/2151046 TalendSolutionExpert 2024-02-06T15:10:57Z