TPigCode component does not work correctly it seems
I am trying to sessionize weblog data using external component datafu using a jar (I am using Talend Enterprise Big Data offering). Below is my TPigCode component code:
register /root/etl/lib/datafu-0.0.9.jar
%declare TIME_WINDOW 5m
define Sessionize datafu.pig.sessions.Sessionize('$TIME_WINDOW');
views = GROUP tPigLoad_1_RESULT BY $1;
tPigCode_1_RESULT = FOREACH views {
visits = ORDER views BY $0;
GENERATE FLATTEN(Sessionize()) AS ($0,$1,$2,$3,$4,session_id);
}
The job compiles but during execution I get the error below. Am I doing something wrong or does Talend not support complex Pig code? I had similar strange errors with REGEX_EXTRACT_ALL function and I had to rewrite my job do do things differently. It seems that TPigCode component does not support advanced features of pig scripting. Also, id there a better documentation out there that tell exactly what is supported in TPigCode Talend component and what is not? Can someone get back to me on this issue!
13/05/31 12:20:21 INFO executionengine.HExecutionEngine: Connecting to hadoop file system at: hdfs://153.64.117.162:8020
13/05/31 12:20:22 INFO executionengine.HExecutionEngine: Connecting to map-reduce job tracker at: HDNameNode:50300
Error during parsing. <line 2, column 0> Syntax error, unexpected symbol at or near 'register'
Caused by:
Failed to parse: <line 2, column 0> Syntax error, unexpected symbol at or near 'register'
Thanks,
Marcin
Hi,
The PigCode components is a component which can be used between a tPigLoad and a tPigStoreResult within the flow. Moreover a tPigCode can register only 1 Pig Latin statement against the Pig server at a time. That means you would need to chain multiple tPigCode.
In order to register a JAR within a PIG jb, you will need to use the hadoop properties in the advanced settings of the tPigLoad: --> Key: "REGISTER" --> Value: "/root/etl/lib/datafu-0.0.9.jar"
