Note: This content is accessible to all versions of every browser. However, this browser does not seem to support current Web standards, preventing the display of our site's design details.


Online outlier detection and removal


P.H. Menold, R. Pearson, F. Allgöwer

Mediterranean Conference on Control and Automation, Haifa, Israel

Outliers occur regularly enough in real-world measurement data to constitute a significant practical problem that is not adequately addressed by traditional smoothing filters designed to reduce the effects of high-frequency noise. To address this problem, this paper describes a simple data cleaning filter for outlier detection and removal which is based on a causal moving data window that is appropriate to real--time applications like closed loop control. This filter is an extension of the well--known median filter: the observed data point $y_k$ is compared to the median $y_k^{dagger}$ of present and past data points. If the distance between these points is large relative to a specified threshold, $y_k$ is declared an outlier and replaced with a more reasonable value $y_k^{star}$. In the most favorable circumstances alters the above described data cleaning filter only outliers (e.g., shot noise) and does not modify nominal data points. Simple implementations of this filter require few tuning parameters and no explicit process model is required for filter tuning. This paper presents some useful tuning guidelines based on simple characterizations of the nominal variation seen in outlier-free portions of the data. To illustrate the utility of this filter, applications are presented for both real data examples and a simulation example where the exact results are known and performance can be assessed more precisely. It is also demonstrated that the data cleaning filter described here can be combined with traditional linear smoothing filters to achieve both protection against outliers and effective noise reduction, but the outlier filter should preceed the noise filter to achieve these results.


Type of Publication:


No Files for download available.
% No recipe for automatically generating a BibTex entry for (06)Talk
Permanent link