python - Remove jumps like peaks and steps in timeseries

Question

I have quite a few sensors in the field that measure water pressure. In the past the height of these sensors have been changed quite a few times creating jumps in the timeseries. Since these timeseries are continuous and I have a manual measurement I should technically be able to remove the jumps (by hand this is easy, but there are too many measurements so I need to do it in python).

I've tried removing the jumps using a median filter but this doesn't really work.

My code:

    # filter out noise in signal (peaks)
    minimumPeak = 0.03 # filter peaks larger than 0.03m
    filtered_value = np.array(im.median_filter(data['value'], 5))
    noise = np.array((filtered_value-data['value']).abs() > minimumPeak)
    data.loc[noise, 'value'] = filtered_value[noise]

data is pandas dataframe containing two columns: 'datetime' and 'value'.

I've also tried to do this manually and got it working in a simple case, but not well in any other. Any idea how I would filter out the jumps?

An example is shown in the picture below (yellow indicating the jumps, red the measurement by hand (it is very well possible that this measurement is not in the beginning as it is in this example))

Answer 1

You have sharp peaks and steps in your data. I guess you want to

• remove the peaks and replace by some averaged values
• remove the steps by cumulative changing the offset of the remaining data values

That's in line with what you said in your last comment. Please note, that this will alter (shift) big parts of your data!

It's important to recognize that the width of both, peaks and steps, is one pixel in your data. Also you can handle both effects pretty much independently.

I suggest to first remove the peaks, then remove the steps.

1. Remove peaks by calculating the absolute difference to the previous and to the next data value, then take the minimum of both, i.e. if your data series is y(i) compute p(i)=min(abs(y(i)-y(i-1)), abs(y(i+1)-y(i))). All values above a threshold are peaks. Take them and replace the data values with the mean of the previous and the next pixel like.

2. Now remove the steps, again by looking for absolute differences of consecutive values (as suggested in the comment by AreTor), s(i)=abs(y(i)-y(i-1)) and look for values above a certain threshold. The positions are the step positions. Create an zero-valued offset array of the same size, then insert the differences of the data points (without the absolute value), then form the cumulative sum and subtract the result from the original data to remove the steps.

Please note that this removes peaks and steps which go up as well as down. If you want to remove only one kind, just don't take the absolute value.