topic Re: How to get next record's value while processing an input ? in Talend Studio

How to get next record's value while processing an input ?

Anonymous — Sat, 16 Nov 2024 09:41:40 GMT

Re: How to get next record's value while processing an input ?

Anonymous — Fri, 02 Jun 2017 10:45:53 GMT

I believe this tutorial will help with this.....

https://www.rilhia.com/quicktips/quick-tip-compare-row-value-against-value-previous-row

If you want to "look ahead" you simply adjust this slightly to hold processing the first row until the second has arrived. Thus allowing you to appear to "look ahead"

Re: How to get next record's value while processing an input ?

Anonymous — Fri, 02 Jun 2017 11:05:05 GMT

Can you please explain it in detail ? how is it possible to get the next row ?

Thanks ,

Praveen.

Re: How to get next record's value while processing an input ?

Anonymous — Fri, 02 Jun 2017 11:09:30 GMT

It is not possible for the current row to get the "next" as such. So what I am suggesting is using a variation on what that tutorial shows.

When your job reads the first row, it is essentially stored in memory. Then, when the second row arrives, the first row does what it needs to with the second row's data, is released and then the second row is stored in memory. This goes on throughout the job until you get to the end of the data set.

This is enabled by the fact that the tMap variables are processed from top to bottom and are held between data rows. Using this functionality (as described in the tutorial....you can literally follow it and recreate it in an example job to see it happening), you can do this inter-row calculation.

Re: How to get next record's value while processing an input ?

Anonymous — Fri, 02 Jun 2017 11:30:43 GMT

In my scenario , current row's data will be modified as per next row's data .So as per your suggestion if i store the first row data in a variable and wait for the second row to arrive , i can compare that with the variable value but how can i edit the previous row data based on this comparison ?

Thanks ,

Praveen .

Re: How to get next record's value while processing an input ?

Anonymous — Fri, 02 Jun 2017 11:59:56 GMT

The previous row's data is in memory. You can edit it in memory as well.

When you output the variables that the row columns are stored in, you can modify the data there....or do it in another tMap variable.

This isn't *easy*, but it is possible and quite logical when you get your head into it. Try it out. The best way to learn something like this is to test it in an example job.

Re: How to get next record's value while processing an input ?

Anonymous — Wed, 01 Nov 2017 10:41:23 GMT

HI Rhall,

I referred the link you have provided but getting error.

My requirement is to store previous record hashkey and price value in two variables. if hashkey is same calculate price difference as below

input

hash key | postcode | Price

1000 abc 100,000

1000 abc 200,000

1000 abc 250,000

1001 xyz 50,000

1001 xyz 250,000

expected:

hash key | postcode | Price | profitmade

1000 | abc | 100,000 | 0

1000 | abc | 200,000 | 100,000

1000 | abc | 250,000 | 50,000

1001 | xyz | 50,000 | 0

1001 | xyz | 250,000 | 200,000

Error I am getting:

org.talend.designer.runprocess.ProcessorException: Job compile errors
At least job "testfile" has a compile errors, please fix and export again.
Error Line: 1275
Detail Message: v_Prev_Price cannot be resolved to a variable

I have attached the print screen of job.

variable v_price expression:

(v_PrevHash!=null && v_PrevHash == row1.hashkey)?(row1.Price-v_Prev_Price):row1.Price

Can you please check what could be the issue?

I also searched and found that there is component 'tMemorizeRows' I am not able find this component in my talend studio.

'tMemorizeRows' : https://help.talend.com/reader/mjoDghHoMPI0yuyZ83a13Q/0oEBI6YCTFXSgc5Z9Gpr6Q

Screen Shot 2017-11-01 at 10.34.57.png

Re: How to get next record's value while processing an input ?

Anonymous — Wed, 01 Nov 2017 12:05:10 GMT

OK, when using tMap variables, they are referred to using Var. before the name. So your variables should be named like this Var.v_Prev_Price, for example. You can drag and drop your tMap variables to another tMap variable expression if you like. That should show you how the names should be formatted. This is almost certainly the cause of your Java error.

Re: How to get next record's value while processing an input ?

TRF — Wed, 01 Nov 2017 12:19:24 GMT

Hi,

In my opinion, the simplest is to use a tJavaRow instead of a tMap for this case.

It should look like this:

// Initialize output_row fields from input_row fields values
output_row.hashkey = input_row.hashkey;
output_row.postcode = input_row.postcode;
output_row.price = input_row.price;

// Set ProfitMade depending on comparison result of previous and current hashkey
if((Integer)globalMap.get("PrevHash") != null && (Integer)globalMap.get("PrevHash") == input_row.hashkey)
	output_row.ProfitMade = input_row.Price - (Integer)globalMap.get("PrevPrice");
else
	output_row.ProfitMade = 0;

// Set the global variables values for next iteration
globalMap.set("PrevHash", input_row.hashkey);
globalMap.set("PrevPrice", input_row.Price);

Check field names and datatypes (integer vs float) before to try this in your job.

Hope this helps.

Re: How to get next record's value while processing an input ?

Anonymous — Wed, 01 Nov 2017 14:25:27 GMT

Thank rhall, I added 'Var' before using variable and it worked. but facing datatype incompatible issue now.

hashkey field data is being generated using MD5 function(which is basically hexadecimal). My source is file. talend reading hashkey as string and giving below error datatype defined in talend is int ( I tried changing datatype as int, long etc)

[statistics] connecting to socket on port 3446

[statistics] connected

For input string: "hashkey"

For input string: "ddbfcac5d5bb83f97906b3326e4dfd97"

[statistics] disconnected

Job testfile ended at 14:17 01/11/2017. [exit code=0]

Re: How to get next record's value while processing an input ?

Anonymous — Wed, 01 Nov 2017 23:37:19 GMT

Can you show some screenshots of your job configuration, since I do not understand what you are trying to do. Are you reading in an MD5 String? I'll need a bit more info

Re: How to get next record's value while processing an input ?

Anonymous — Wed, 09 May 2018 23:02:58 GMT

In a similar situation:

Input

MASTER_CODE	COMM
XLMA	C-123-1
AAAA	A-111-1
BBBB	C-123-1
BBBB	D-123-1
CCCC	C-333-3
XLMS-A	C-553-2
XLMS-A	D-323-1
XLMS-A	E-311-3

Desired output

MASTER_CODE	COMM
XLMA	C-123-1
AAAA	A-111-1
BBBB	C-123-1
	D-123-1
CCCC	C-333-3
XLMS-A	C-553-2
	D-323-1
	E-311-3

I'm trying to do it using tJavaRow but no luck, this is the code I'm using:

row2.MASTER_CODE.equals(row1.MASTER_CODE);
row2.COMM.equals(row1.COMM);

if((String)globalMap.get("PrevRow").equals(row1.MASTER_CODE))
row2.MASTER_CODE.equals("");
else
row2.MASTER_CODE.equals(row1.MASTER_CODE);
globalMap.set("PrevRow", row1.MASTER_CODE);

Re: How to get next record's value while processing an input ?

Anonymous — Thu, 10 May 2018 09:16:33 GMT

Your Java is incorrect. The "equals" method doesn't assign a value, it checks to see if two values are equal. This should work....

if(globalMap.get("PrevRow")!=null&&((String)globalMap.get("PrevRow")).equals(row1.MASTER_CODE)){
      row2.MASTER_CODE = "";
}else{
     globalMap.put("PrevRow",row1.MASTER_CODE);
     row2.MASTER_CODE = ((String)globalMap.get("PrevRow"));
     row2.COMM = row1.COMM;
}

Re: How to get next record's value while processing an input ?

Anonymous — Thu, 10 May 2018 20:25:55 GMT

Thanks for your reply!
This almost solves it. The remaining issue is that the COMM field is not showing the correct values:

MASTER_CODE	COMM	Should be
XLMA	C-123-1
AAAA	A-111-1
BBBB	C-123-1
	C-123-1	D-123-1
CCCC	C-333-3
XLMS-A	C-553-2
	C-553-2	D-323-1
	C-553-2	E-311-3

Re: How to get next record's value while processing an input ?

TRF — Thu, 10 May 2018 21:19:52 GMT

What about this one?

if((String)globalMap.get("PrevRow") != null &&
((String)globalMap.get("PrevRow")).equals(input_row.MASTER_CODE)){
output_row.MASTER_CODE = "";
}else{
globalMap.put("PrevRow", input_row.MASTER_CODE);
output_row.MASTER_CODE = input_row.MASTER_CODE;
}
output_row.COMM = input_row.COMM;

Re: How to get next record's value while processing an input ?

Anonymous — Thu, 10 May 2018 23:10:01 GMT

Excellent, this worked! Simply moving the COMM field out of the IF statement solves it.

Thanks for your help!

Re: How to get next record's value while processing an input ?

Anonymous — Fri, 11 May 2018 09:11:03 GMT

Sorry, I wrote that straight to the forum without testing it. I meant the COMM field to be outside of the IF hence it was only included once.

Re: How to get next record's value while processing an input ?

Fri, 22 Jun 2018 11:22:18 GMT

i also need to get next record's value for process. i didn't find the solution in forum. kindly share the solution u got

Re: How to get next record's value while processing an input ?

Anonymous — Tue, 18 Sep 2018 08:32:01 GMT

How do you decide the next row , it should be based on a column. for example , for a customer-purchase information file i would consider that the next row is based upon his purchase date ie , for a customer A (considering two purchases for him) the next row will be the row having the maximum date and the current row will be the row having minimum date. So the next and the current rows are always dependent upon a column (date,sequential number,rank etc.)

Hence , for the above example ,sorting descending upon the purchase-date column on every customer group , we can get the next purchase date for the customer row with current purchase date by fetching the previous date value using tmemorize or a variable storing the previous value.

In Java , it is always a top down approach and we cant get the next iteration's value sitting on the current iteration . The above solution would be a workaround to achieve what you want.