Bay-Delta SELFE Tools

error_detect Module

error_detect.med_outliers(ts, level=4, scale=None, filt_len=7, range=(None, None), quantiles=(0.25, 0.75), copy=True)

Detect outliers by running a median filter, subtracting it from the original series and comparing the resulting residuals to a global robust range of scale (the interquartile range). Individual time points are rejected if the residual at that time point is more than level times the range of scale.

The original concept comes from Basu & Meckesheimer (2007) although they didn’t use the interquartile range but rather expert judgment. To use this function effectively, you need to be thoughtful about what the interquartile range will be. For instance, for a strongly tidal flow station it is likely to

level: Number of times the scale or interquantile range the data has to be
to be rejected.
scale: Expert judgment of the scale of maximum variation over a time step.
If None, the interquartile range will be used. Note that for a strongly tidal station the interquartile range may substantially overestimate the reasonable variation over a single time step, in which case the filter will work fine, but level should be set to a number (less than one) accordingly.

filt_len: length of median filter, default is 5

quantiles : tuple of quantiles defining the measure of scale. Ignored
if scale is given directly. Default is interquartile range, and this is almost always a reasonable choice.

copy: if True, a copy is made leaving original series intact

You can also specify rejection of values based on a simple range

Returns: copy of series with outliers replaced by nan