[resolved] Can't filter out Rows

NPreuschoff — Sat, 16 Nov 2024 10:24:15 GMT

Hi!
I'm trying to create a new component to automatical filter out all empty rows.
This component also serves as a learning experience for me so it's my first own component.
My Goal is to have a component which i can just drop into the flow to filter out the empty rows.
I know that this can be achieved with tFilterRow but i want a component which does it without any extra work.
At the moment the flow must have only variables of type String. I will implement the detection of the type later when the filtering works correctly.

My problem is that if i don't write the content of input_row to the output_row, Talend automatically writes the values from the row before into the output_row instead.
I got it working by putting a continue into the if-statement but this only works correct for flows that are just a single line and surely isn't the right way to do it.

I tried to write the rejected rows into a second output instead but it didn't work. This just creates an empty output with the same rowcount as the input and doesn't change the filter output at all
Am i missing something? I read on a tutorial from powerupbi that i need DATA_AUTO_REPLICATE="false" in my headers but if i put it there the component can't be loaded and the Talendlog says, that DATA_AUTO_REPLICATE isn't allowed in the headers
I do have HAS_CONDITIONAL_OUTPUTS="true" in my Headers
I'm using Talend Data Fabric 6.2.1

Regards,
Nils Preuschoff
tFilterEmptyRows_begin.javajet

<%@ jet
    imports="
        org.talend.core.model.process.INode
        org.talend.core.model.process.ElementParameterParser
        org.talend.core.model.metadata.IMetadataTable
        org.talend.core.model.metadata.IMetadataColumn
        org.talend.core.model.process.IConnection
        org.talend.core.model.process.IConnectionCategory
        org.talend.designer.codegen.config.CodeGeneratorArgument
        org.talend.core.model.metadata.types.JavaTypesManager
        org.talend.core.model.metadata.types.JavaType
        java.util.List
        java.util.Map      
    "
%>
<%
    CodeGeneratorArgument codeGenArgument = (CodeGeneratorArgument) argument;
    INode node = (INode)codeGenArgument.getArgument();
    String cid = node.getUniqueName(); 
%> 

int nb_line_<%=cid%> = 0;
int nb_line_empty_<%=cid%> = 0;
int nb_line_ok_<%=cid%> = 0;

tFilterEmptyRows_main.javajet

<%@ jet
    imports="
        org.talend.core.model.process.INode
        org.talend.core.model.process.ElementParameterParser
        org.talend.core.model.metadata.IMetadataTable
        org.talend.core.model.metadata.IMetadataColumn
        org.talend.core.model.process.IConnection
        org.talend.core.model.process.IConnectionCategory
        org.talend.designer.codegen.config.CodeGeneratorArgument
        org.talend.core.model.metadata.types.JavaTypesManager
        org.talend.core.model.metadata.types.JavaType
        java.util.List
        java.util.Map      
    "
%>
<%
    CodeGeneratorArgument codeGenArgument = (CodeGeneratorArgument) argument;
    INode node = (INode)codeGenArgument.getArgument();
	String cid = node.getUniqueName(); 
%>
	boolean empty = true; <%
	List< ? extends IConnection> incoming = node.getIncomingConnections();
	if ((incoming != null) && (incoming.size() > 0)) {//A
		IConnection income2 = incoming.get(0);
		IMetadataTable income = income2.getMetadataTable();
		if (income != null) {//B
		  	List<IMetadataColumn> columns = income.getListColumns();  
		  	for( IMetadataColumn col : income.getListColumns() ) { %> 
	
		  		
				if (<%=income2.getName()%>.<%= col%> != null && !(<%=income2.getName()%>.<%= col%>.trim().isEmpty())) { //D
					
					empty  = false;
				}//D
				<%
			}%>  
		   
		   if(empty) {//E
		   		nb_line_empty_<%=cid %>++;	 
				continue;
		   		
		   } else {
		   	<%
		   	
		   		List< ? extends IConnection> filter_conns = node.getOutgoingConnections("FILTER");
		   		if ((filter_conns != null) && (filter_conns.size() > 0)) {//F
					IConnection metadata = filter_conns.get(0);%>
				 	<%=metadata.getName()%> = new <%= metadata.getName() %>Struct();
					<%
					for(IMetadataColumn col : income.getListColumns() ) {//G
						%>
					
						<%=metadata.getName()%>.<%=col %> = <%= income2.getName() %>.<%= col %>;
						
						<%
					}//G	
		   		}//F
		   				%>
		   		nb_line_ok_<%=cid%>++;
		   }//E
		  <% 
		 }//B
	}//A
    %>
    
    nb_line_<%=cid %>++;
    globalMap.put("<%=cid%>_NB_LINE",nb_line_<%=cid%>);
    globalMap.put("<%=cid%>_NB_LINE_EMPTY",nb_line_empty_<%=cid%>);
    globalMap.put("<%=cid%>_NB_LINE_OK",nb_line_ok_<%=cid%>);

tFilterEmptyRows_java.xml

<COMPONENT>
  <HEADER
    PLATEFORM="ALL"
    SERIAL=""
    VERSION="0.1"
    STATUS="ALPHA"
  
    COMPATIBILITY="ALL"
    AUTHOR="Nils Preuschoff"
    RELEASE_DATE="20160829A"
    SCHEMA_AUTO_PROPAGATE="true"
    HAS_CONDITIONAL_OUTPUTS="true"
    DATA_AUTO_PROPAGATE="false"
    STARTABLE="false"
    PARTITIONING="AUTO"
	LOG4J_ENABLED="true"
  >
    <SIGNATURE/>
  </HEADER>
  
  <FAMILIES>
    <FAMILY>Quinscape</FAMILY>
  </FAMILIES>
  
  <DOCUMENTATION>
    <URL/>
  </DOCUMENTATION>
  
  <CONNECTORS>
    <CONNECTOR CTYPE="FLOW" MAX_INPUT="1" MAX_OUTPUT="0"/>
	<CONNECTOR NAME="FILTER" MAX_OUTPUT="1" CTYPE="FLOW" BASE_SCHEMA="FLOW" COLOR="086438" />
    <CONNECTOR CTYPE="SUBJOB_OK" MAX_INPUT="1" />
    <CONNECTOR CTYPE="SUBJOB_ERROR" MAX_INPUT="1" />
    <CONNECTOR CTYPE="COMPONENT_OK" />
    <CONNECTOR CTYPE="COMPONENT_ERROR" />
    <CONNECTOR CTYPE="RUN_IF" />
  </CONNECTORS>
  
  <PARAMETERS>
		<PARAMETER NAME="SCHEMA_FILTER" FIELD="SCHEMA_TYPE" CONTEXT="FILTER"  NUM_ROW="1"  READONLY="true"/>
	
  </PARAMETERS>
  
  <CODEGENERATION/>
  
  <RETURNS>
    <RETURN NAME="NB_LINE" TYPE="id_Integer" AVAILABILITY="AFTER"/>
    <RETURN NAME="NB_LINE_OK" TYPE="id_Integer" AVAILABILITY="AFTER"/>
    <RETURN NAME="NB_LINE_EMPTY" TYPE="id_Integer" AVAILABILITY="AFTER"/>
  </RETURNS>
  
</COMPONENT>

Re: [resolved] Can't filter out Rows

Anonymous — Mon, 19 Sep 2016 08:44:22 GMT

Hi,
Have you already checked article about:TalendHelpCenter:How to create a custom component?
Best regards
Sabrina

Re: [resolved] Can't filter out Rows

NPreuschoff — Thu, 22 Sep 2016 08:26:47 GMT

Hi Sabrina,
sorry for the late response here.
Yes i did read it and also this one
I still can't find my problem with this.
I can add the content of the other files, too if that helps
Best regards
Nils

Re: [resolved] Can't filter out Rows

NPreuschoff — Thu, 29 Dec 2016 10:16:06 GMT

After i had some time again to look at this i finally found a solution
I just needed to set the outputrow to null if the input is empty

Re: [resolved] Can't filter out Rows

Anonymous — Fri, 30 Dec 2016 06:31:03 GMT

Hi NPreuschoff,
Thanks for posting that you have resolved this issue by yourself and sharing your solution here.
Best regards
Sabrina

topic Re: [resolved] Can't filter out Rows in Talend Studio

[resolved] Can't filter out Rows

Re: [resolved] Can't filter out Rows

Re: [resolved] Can't filter out Rows

Re: [resolved] Can't filter out Rows

Re: [resolved] Can't filter out Rows