Fast window correlations over uncooperative time series

Richard Cole, Dennis Shasha, Xiaojian Zhao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Data arriving in time order (a data stream) arises in fields including physics, finance, medicine, and music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. Further, the number of sensors is increasing, so correlating data between sensors becomes ever more critical in order to distill knowlege from the data. In many applications such as finance, recent correlations are of far more interest than long-term correlation, so correlation over sliding windows (windowed correlation) is the desired operation. Fast response is desirable in many applications (e.g., to aim a telescope at an activity of interest or to perform a stock trade). These three factors - data size, windowed correlation, and fast response - motivate this work. Previous work [10, 14] showed how to compute Pearson correlation using Fast Fourier Transforms and Wavelet transforms, but such techniques don't work for time series in which the energy is spread over many frequency components, thus resembling white noise. For such "uncooperative" time series, this paper shows how to combine several simple techniques - sketches (random projections), convolution, structured random vectors, grid structures, and combinatorial design - to achieve high performance windowed Pearson correlation over a variety of data sets.

Original languageEnglish (US)
Title of host publicationKDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
EditorsR.L. Grossman, R. Bayardo, K. Bennett, J. Vaidya
Pages743-749
Number of pages7
DOIs
StatePublished - 2005
EventKDD-2005: 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - Chicago, IL, United States
Duration: Aug 21 2005Aug 24 2005

Other

OtherKDD-2005: 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
CountryUnited States
CityChicago, IL
Period8/21/058/24/05

Fingerprint

Time series
Sensors
Finance
Medicine
Physics
White noise
Convolution
Telescopes
Fast Fourier transforms
Wavelet transforms

Keywords

  • Correlation
  • Randomized algorithms
  • Time series

ASJC Scopus subject areas

  • Information Systems

Cite this

Cole, R., Shasha, D., & Zhao, X. (2005). Fast window correlations over uncooperative time series. In R. L. Grossman, R. Bayardo, K. Bennett, & J. Vaidya (Eds.), KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 743-749) https://doi.org/10.1145/1081870.1081966

Fast window correlations over uncooperative time series. / Cole, Richard; Shasha, Dennis; Zhao, Xiaojian.

KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ed. / R.L. Grossman; R. Bayardo; K. Bennett; J. Vaidya. 2005. p. 743-749.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cole, R, Shasha, D & Zhao, X 2005, Fast window correlations over uncooperative time series. in RL Grossman, R Bayardo, K Bennett & J Vaidya (eds), KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 743-749, KDD-2005: 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, United States, 8/21/05. https://doi.org/10.1145/1081870.1081966
Cole R, Shasha D, Zhao X. Fast window correlations over uncooperative time series. In Grossman RL, Bayardo R, Bennett K, Vaidya J, editors, KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2005. p. 743-749 https://doi.org/10.1145/1081870.1081966
Cole, Richard ; Shasha, Dennis ; Zhao, Xiaojian. / Fast window correlations over uncooperative time series. KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. editor / R.L. Grossman ; R. Bayardo ; K. Bennett ; J. Vaidya. 2005. pp. 743-749
@inproceedings{93ce1b45994944c2b7db48fefa4b842e,
title = "Fast window correlations over uncooperative time series",
abstract = "Data arriving in time order (a data stream) arises in fields including physics, finance, medicine, and music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. Further, the number of sensors is increasing, so correlating data between sensors becomes ever more critical in order to distill knowlege from the data. In many applications such as finance, recent correlations are of far more interest than long-term correlation, so correlation over sliding windows (windowed correlation) is the desired operation. Fast response is desirable in many applications (e.g., to aim a telescope at an activity of interest or to perform a stock trade). These three factors - data size, windowed correlation, and fast response - motivate this work. Previous work [10, 14] showed how to compute Pearson correlation using Fast Fourier Transforms and Wavelet transforms, but such techniques don't work for time series in which the energy is spread over many frequency components, thus resembling white noise. For such {"}uncooperative{"} time series, this paper shows how to combine several simple techniques - sketches (random projections), convolution, structured random vectors, grid structures, and combinatorial design - to achieve high performance windowed Pearson correlation over a variety of data sets.",
keywords = "Correlation, Randomized algorithms, Time series",
author = "Richard Cole and Dennis Shasha and Xiaojian Zhao",
year = "2005",
doi = "10.1145/1081870.1081966",
language = "English (US)",
pages = "743--749",
editor = "R.L. Grossman and R. Bayardo and K. Bennett and J. Vaidya",
booktitle = "KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining",

}

TY - GEN

T1 - Fast window correlations over uncooperative time series

AU - Cole, Richard

AU - Shasha, Dennis

AU - Zhao, Xiaojian

PY - 2005

Y1 - 2005

N2 - Data arriving in time order (a data stream) arises in fields including physics, finance, medicine, and music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. Further, the number of sensors is increasing, so correlating data between sensors becomes ever more critical in order to distill knowlege from the data. In many applications such as finance, recent correlations are of far more interest than long-term correlation, so correlation over sliding windows (windowed correlation) is the desired operation. Fast response is desirable in many applications (e.g., to aim a telescope at an activity of interest or to perform a stock trade). These three factors - data size, windowed correlation, and fast response - motivate this work. Previous work [10, 14] showed how to compute Pearson correlation using Fast Fourier Transforms and Wavelet transforms, but such techniques don't work for time series in which the energy is spread over many frequency components, thus resembling white noise. For such "uncooperative" time series, this paper shows how to combine several simple techniques - sketches (random projections), convolution, structured random vectors, grid structures, and combinatorial design - to achieve high performance windowed Pearson correlation over a variety of data sets.

AB - Data arriving in time order (a data stream) arises in fields including physics, finance, medicine, and music, to name a few. Often the data comes from sensors (in physics and medicine for example) whose data rates continue to improve dramatically as sensor technology improves. Further, the number of sensors is increasing, so correlating data between sensors becomes ever more critical in order to distill knowlege from the data. In many applications such as finance, recent correlations are of far more interest than long-term correlation, so correlation over sliding windows (windowed correlation) is the desired operation. Fast response is desirable in many applications (e.g., to aim a telescope at an activity of interest or to perform a stock trade). These three factors - data size, windowed correlation, and fast response - motivate this work. Previous work [10, 14] showed how to compute Pearson correlation using Fast Fourier Transforms and Wavelet transforms, but such techniques don't work for time series in which the energy is spread over many frequency components, thus resembling white noise. For such "uncooperative" time series, this paper shows how to combine several simple techniques - sketches (random projections), convolution, structured random vectors, grid structures, and combinatorial design - to achieve high performance windowed Pearson correlation over a variety of data sets.

KW - Correlation

KW - Randomized algorithms

KW - Time series

UR - http://www.scopus.com/inward/record.url?scp=32344446365&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=32344446365&partnerID=8YFLogxK

U2 - 10.1145/1081870.1081966

DO - 10.1145/1081870.1081966

M3 - Conference contribution

AN - SCOPUS:32344446365

SP - 743

EP - 749

BT - KDD-2005 - Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

A2 - Grossman, R.L.

A2 - Bayardo, R.

A2 - Bennett, K.

A2 - Vaidya, J.

ER -