How to implement MD5 mechanism ?

Anonymous — Sat, 16 Nov 2024 10:21:37 GMT

I have a set of csv files which load in daily but do not have an audit date columns and a Primary key as well to identify a new record.
I want to implement MD5 mechanism to help distinguish the new records. I am using the Talend Big Data v6.2
I tried to implement the logic given in this link : https://community.talend.com/t5/Design-and-Development/Getting-the-MD5-hash-of-a-database-record-field/m-p/71553
and also imported tMD5Sum.zip
When I import the routine using (Select archive file option) it says 'No valid item to import'.
I copy pasted the syntax from tMD5Sum_main.javajet file but to no avail.

Is there any way I can link this routine from the exchange and pull it direcly to my workspace or
is there any easy way out to solve a problem of this sort without implementing the Hash key mechanism?
My target dB is Cassandra 2.0.

Code :

<%@ jet

imports="

org.talend.core.model.process.INode

org.talend.core.model.process.ElementParameterParser

org.talend.core.model.metadata.IMetadataTable

org.talend.core.model.metadata.IMetadataColumn

org.talend.core.model.process.IConnection

org.talend.core.model.process.IConnectionCategory

org.talend.designer.codegen.config.CodeGeneratorArgument

java.util.List

org.talend.core.model.metadata.types.JavaTypesManager

org.talend.core.model.metadata.types.JavaType

"

%>

<%

CodeGeneratorArgument codeGenArgument = (CodeGeneratorArgument) argument;

INode node = (INode)codeGenArgument.getArgument();

String cid = node.getUniqueName();

String fileName = ElementParameterParser.getValue(node, "__FILENAME__");

String outputConnName = null;

List< ? extends IConnection> conns = node.getOutgoingSortedConnections();

if (conns!=null) {

if (conns.size()>0) {

for (int i=0;i<conns.size();i++) {

IConnection connTemp = conns.get(i);

if (connTemp.getLineStyle().hasConnectionCategory(IConnectionCategory.DATA)) {

outputConnName = connTemp.getName();

break;

}

}

}

}

if (outputConnName != null){

%>

<%=outputConnName %> = new <%=outputConnName %>Struct();

<%=outputConnName %>.filename = <%=fileName %>;

<%=outputConnName %>.md5_sum = com.ostermiller.util.MD5.getHashString(new java.io.File(<%=fileName %>));

<%

}

%>

Re: How to implement MD5 mechanism ?

Anonymous — Mon, 10 Oct 2016 07:36:44 GMT

Hi,
Are you looking for a custom component tMD5Sum on talend exchange portal?
https://exchange.talend.com/#marketplaceproductoverview:gallery=marketplace%252F1&pi=marketplace%252F1%252Fproducts%252F524%252Fitems%252F774
Have you tried to install it into studio to see if it works?
Here is a KB article about: TalendHelpCenter:Installing a custom component
Best regards
Sabrina

Re: How to implement MD5 mechanism ?

Anonymous — Tue, 11 Oct 2016 11:12:02 GMT

thank you , it worked like a charm,
Are there any solved examples on tMD5Sum component as I couldnt find any online.

topic Re: How to implement MD5 mechanism ? in Talend Studio

How to implement MD5 mechanism ?

Re: How to implement MD5 mechanism ?

Re: How to implement MD5 mechanism ?