https://community.qlik.com/t5/Talend-Studio/S-ark-Streaming-creating-multiple-hdfs-directory/m-p/2303698#M75545 <P>You can use tHiveoutput component to append to the same directory, which invokes df.append method in partitioned manner.One way i could think of.</P><P>&nbsp;</P> Wed, 07 Mar 2018 00:39:23 GMT Anonymous 2018-03-07T00:39:23Z https://community.qlik.com/t5/Talend-Studio/S-ark-Streaming-creating-multiple-hdfs-directory/m-p/2303697#M75544 <P>Hi All,</P><P>&nbsp;</P><P>I have a spark streaming job which is consuming messages from MapR stream. I am trying to put the messages into HDFS location and from there I am trying to process it using a batch process.&nbsp;</P><P>The problem is every batch (I have set&nbsp; a 2 minutes batch) in the streaming job is creating a separate&nbsp;directory in HDFS&nbsp;with a timestamp value. I am not sure how to merge all the files for a particular day and feed the merged file to my end of day batch job to be processed.</P><P>&nbsp;</P><P>Can anybody please help here?</P><P>&nbsp;</P> Sat, 16 Nov 2024 08:35:44 GMT https://community.qlik.com/t5/Talend-Studio/S-ark-Streaming-creating-multiple-hdfs-directory/m-p/2303697#M75544 Anonymous 2024-11-16T08:35:44Z https://community.qlik.com/t5/Talend-Studio/S-ark-Streaming-creating-multiple-hdfs-directory/m-p/2303698#M75545 <P>You can use tHiveoutput component to append to the same directory, which invokes df.append method in partitioned manner.One way i could think of.</P><P>&nbsp;</P> Wed, 07 Mar 2018 00:39:23 GMT https://community.qlik.com/t5/Talend-Studio/S-ark-Streaming-creating-multiple-hdfs-directory/m-p/2303698#M75545 Anonymous 2018-03-07T00:39:23Z