Anomaly detection models for IoT time series data

Authors

  • Federico Giannoni
  • Marco Mancini
  • Federico Giannoni
  • Federico Marinelli

Keywords:

time series data, Wireless Sensor Networks, anomaly detection

Abstract

Our experiments yielded very diverse results. In general, it is hard to identify which algorithm is better than the others overall, due to the fact that some perform better in identifying single outliers, while others are better in identification of anomalous trends. For this reason, an optimal solution could only be obtained by selecting a few of the proposed solutions to form a model based on an ensemble of experts. The experts’ outputs would then be combined using either a majority vote approach, or a weight-based strategy, to decide which acquisition is to be classified as anomalous. Judging from our experiments, there are many reasonable combinations of experts. For instance, one may use two separate instances of the low-high pass filter, one tuned to detect single anomalies and one tuned to detect anomalous trends. Alternatively, one could use S-ESD to identify sporadic outliers, as this was the algorithm that performed better given such task and the univariate gaussian predictor to identify anomalous trends. As far as being able to detect sensor faults or damage from detected anomalies, we can only propose our solution, due to the fact that we did not have enough labelled anomalous trends (such as the one covering the period between the 9th and the 11th of November, that was used as test set) to test it. Our idea was to fix an interval t and then use the frequency of events that were usually registered in such interval during anomalous behaviours, to train a linear regressor that would be able to predict a frequency threshold T. Once we learned T, we would consider a sensor damaged whenever it was responsible for more than T detected anomalies in an interval t.

References

J. Tukey, Exploratory Data Analysis, Addison Wesley, MA, 1977.

Y. Yu, Y. Zhu, S. Li, D. Wan, Time Series Outlier Detection Based on Sliding Window Prediction, College of Computer & Information, Hohai University, CN, 2014.

N. Gornitz, M. Braun, M. Kloft, Hidden Markov Anomaly Detection, Berlin Institute of Technology, 2014.

D. Hill, B. Minsker, E. Amir, Real-time Bayesian Anomaly Detection for Environmental Sensor Data, University of Illinois, 2009.

M. Breunig, H. Kriegel, R. Ng, J. Sander, LOF: Identifying Density-Based Local Outliers, ACM SIGMOD, 2000.

Z. He, X. Xu, and S. Deng. Discovering cluster-based local outliers, Pattern Recognition Letters, 2003.

N. Shahid, I. Naqvi, S. Qaisar, One-class support vector machines: analysis of outlier detection for wireless sensor networks in harsh environments, 2013.

P. Malhotra, L. Vig, G. Shroff, P. Agarwal, Long Short Term Memory Networks for Anomaly Detection in Time Series, 2015.

M. Hayes, M. Capretz, Contextual Anomaly Detection in Big Sensor Data, 2014.

J. Hochenbaum, O. Vallis, A. Kejariwal, Automatic Anomaly Detection in the Cloud via Statistical Learning, Twitter Inc., 2017.

R. Cleveland, W. Cleveland, J. McRae, I. Terpenning, STL: A Seasonal-Trend Decomposition Procedure Based on LOESS, Journal of Official Statistics, 1990.

M. Amer, M. Goldstein, Nearest-Neighbor and Clustering based Anomaly Detection Algorithm for RapidMiner, 2011.

Runge Jakob, Sejdinovic Dino. Detecting causal associations in large nonlinear time series datasets, 2017

Downloads

Published

2024-01-04