topic Re: Kafka input file store into HDFS. in Talend Studio

Kafka input file store into HDFS.

Anonymous — Thu, 21 Mar 2019 04:04:33 GMT

Trying to store kafka input file into HDFS.. it is long running. File created into HDFS but content not copied into file. Job is long running. Can please help me what is the things I am doing wrong.

Re: Kafka input file store into HDFS.

vapukov — Thu, 21 Mar 2019 06:46:19 GMT

Hi,

first of all - I suggest you delete connection between Kafaka Connection and KafkaInput

this is 2 independent parts:

everything whaat run before job
and real job

2nd - there are many paramters affected for total performance, but insert each record non stop into HDFS direct might be not. the best idea

try to test in main part of job

create infinite loop:

fetch some limited number of records as a variant 10000 messages
store all to local csv file
append file to HDFS

it could be faster overall

P.S.

but as I mention above - real performance depends on many factors, network latency is one of them

for example:

zip 100 000 2kb files
transfer them to the remote server and unzip all over 1Gb network

will be 50+ times faster than copy files direct, same with databases - import from csv or batch inserts up to 100+ times faster than insert by single row

Re: Kafka input file store into HDFS.

Anonymous — Thu, 21 Mar 2019 08:44:59 GMT

According you suggestion I tried to to store data of kafka topic into csv files or tlogrows because this topic has only 280 records. But both case taking long time.

Re: Kafka input file store into HDFS.

vapukov — Thu, 21 Mar 2019 09:01:30 GMT

if it slow with local file or logRow - you need seriously investigate your network architecture

Kafka extremely fast and no visible bottlenecks in this job

do you test your kafka connection with any other tools? like command line client