But what I didn't explain: I'm not always comparing the same line together
I'm using ODAG to insert selection from my user
if I have 5 lines (1,2,3,4,5) => my user may filters on a date or a category, then I wil have to compare only line 1,3,5
(Right now I'm testing an optimization which consists to compare only required fields and not the 60 fields, in most of the case i have less than 30 fields to compare so I made a table with combination of all fields - it raise to around 400 combinations - then with a for loop, I do the peek only on the required field ... which I hope will speed up the process a bit )
I hope to find a way to do less peek using PurgeChar and subfield on a concatenated string (but I'm not sure it will be faster...)
It's an interesting case and I think it will be quite hard to improve here the performance because it is a comparing of each field and for each record against the previous one. Especially if more as 90% are different to each other it doesn't look very promising to develop a logic to exclude the equal records from the check.
Because of the fact that there are so many checks against the previous value within these loads it might be worth to try previous() instead of peek(). AFAIK both are very similar in the performance but I never measured them. A further measure which may run a bit faster might be to change your check to something like this:
rangesum(-(peek(A)=A), -(peek(B)=B), ...)
A quite different method to your approach could be to transform the data with a crosstable-statement, ordering the result properly in a following load and applying the check only to one field. Of course it would run against much more records and therefore I'm not sure that there would be any improvements - but maybe you apply here some incremental logics and/or pre-calculate some results so that your final ODAG don't need to do the check-work else just to pick the appropriate results.