13 Replies Latest reply: Mar 10, 2017 9:08 AM by frank norrmann RSS

    Filter out sharp spikes (outliers)

    frank norrmann

      I have data that normally follows a very clear and smooth line that can slope upward and downwards but it has a lot of short and very steep spikes that distort the data (from protocol errors). I would like to ignore them in my chart, esentlially "clipping them out" of the data. For example, a typical outliner could be identified in the following data sample:

       

      2418

      2418

      2419

      2418

      5699

      2420

      2418

      2419

       

      or

       

      1418

      1419

      1420

      1421

      3566

      1430

      1432

      1433

       

      The ones in red are the outliers I need to filter out of my chart. There are many of those and their magnitude changes slightly over the dataset, so it is not very straight forward to just use an if/then condition to filter out the ones that are >5600 for example, as it would leave holes in the curve.

       

      An example screenshot of my data is attached

       

      I was hoping to be able to have a filter expression that can ignore them based on the median (or other method).

       

      EDIT: I have added an example screen shot of the data graph with and without a very simple upper boundary IF condition (if([E7 Nivel de producto (mm)]<4000,median([E7 Nivel de producto (mm)])) that helps a bit to cleanup the curve, but it is very limited and "dumb" and will not work well for all the data.

       

      Help is greatly appreciated!