Finding hierarchical heavy hitters in streaming data

Graham Cormode, Flip Korn, S. Muthukrishnan, Divesh Srivastava

    Research output: Contribution to journalArticle

    Abstract

    Data items that arrive online as streams typically have attributes which take values from one or more hierarchies (time and geographic location, source and destination IP addresses, etc.). Providing an aggregate view of such data is important for summarization, visualization, and analysis. We develop an aggregate view based on certain organized sets of large-valued regions (heavy hitters) corresponding to hierarchically discounted frequency counts. We formally define the notion of hierarchical heavy hitters (HHHs). We first consider computing (approximate) HHHs over a data stream drawn from a single hierarchical attribute. We formalize the problem and give deterministic algorithms to find them in a single pass over the input. In order to analyze a wider range of realistic data streams (e.g., from IP traffic-monitoring applications), we generalize this problem to multiple dimensions. Here, the semantics of HHHs are more complex, since a child node can have multiple parent nodes. We present online algorithms that find approximate HHHs in one pass, with provable accuracy guarantees. The product of hierarchical dimensions forms a mathematical lattice structure. Our algorithms exploit this structure, and so are able to track approximate HHHs using only a small, fixed number of statistics per stored item, regardless of the number of dimensions. We show experimentally, using real data, that our proposed algorithms yields outputs which are very similar (virtually identical, in many cases) to offline computations of the exact solutions, whereas straightforward heavy-hitters-based approaches give significantly inferior answer quality. Furthermore, the proposed algorithms result in an order of magnitude savings in data structure size while performing competitively.

    Original languageEnglish (US)
    Article number16
    JournalACM Transactions on Knowledge Discovery from Data
    Volume1
    Issue number4
    DOIs
    StatePublished - Jan 1 2008

      Fingerprint

    Keywords

    • Approximation algorithms
    • Data mining
    • Network data analysis

    ASJC Scopus subject areas

    • Computer Science(all)

    Cite this