topic Re: How to stream log file from server into Hadoop ? Incrementally ? in Talend Studio

How to stream log file from server into Hadoop ? Incrementally ?

gt0731 — Sat, 16 Nov 2024 10:03:52 GMT

We have server Log file data need to push into Hadoop using talend.
We tried :
regex pattern to directly load the server log data into hadoop

thdfsconnection
|
|
tfileinputregex -->thdfsoutput

But it did not work. As it is not outputting anything in target hadoop. Please suggest if we have alternate way to do this.

Attached is console output after job run :
ConsoleOutput.txt_20170219-0358.txt
Any help on it would be much appreciated .

Re: How to stream log file from server into Hadoop ? Incrementally ?

Anonymous — Mon, 06 Mar 2017 08:39:04 GMT

Hi,
For this error " Could not locate executable null\bin\winutils.exe in the Hadoop binaries.", could you please take a look at this Kb article about: TalendHelpCenter:The missing winutils.exe program in the Big Data Jobs to see if this solution is OK with you.
Best regards
Sabrina

Re: How to stream log file from server into Hadoop ? Incrementally ?

gt0731 — Tue, 07 Mar 2017 10:04:22 GMT

Issue is i have to Parse the Log data and Output Into HDFS.