What's new: Finding significant differences in network data streams

Graham Cormode, Shanmugavelayutham Muthukrishnan

    Research output: Contribution to journalArticle

    Abstract

    Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about "what's new." Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the mo st significant deltoids in high-speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.

    Original languageEnglish (US)
    Pages (from-to)1219-1232
    Number of pages14
    JournalIEEE/ACM Transactions on Networking
    Volume13
    Issue number6
    DOIs
    StatePublished - Dec 1 2005

    Fingerprint

    Routers
    Managers
    Monitoring

    Keywords

    • Change detection
    • Data streams
    • Deltoids
    • Network data analysis

    ASJC Scopus subject areas

    • Software
    • Computer Science Applications
    • Computer Networks and Communications
    • Electrical and Electronic Engineering

    Cite this

    What's new : Finding significant differences in network data streams. / Cormode, Graham; Muthukrishnan, Shanmugavelayutham.

    In: IEEE/ACM Transactions on Networking, Vol. 13, No. 6, 01.12.2005, p. 1219-1232.

    Research output: Contribution to journalArticle

    Cormode, Graham ; Muthukrishnan, Shanmugavelayutham. / What's new : Finding significant differences in network data streams. In: IEEE/ACM Transactions on Networking. 2005 ; Vol. 13, No. 6. pp. 1219-1232.
    @article{0cc5eab8d2d44e6a95cad2ea3816db6a,
    title = "What's new: Finding significant differences in network data streams",
    abstract = "Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about {"}what's new.{"} Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the mo st significant deltoids in high-speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.",
    keywords = "Change detection, Data streams, Deltoids, Network data analysis",
    author = "Graham Cormode and Shanmugavelayutham Muthukrishnan",
    year = "2005",
    month = "12",
    day = "1",
    doi = "10.1109/TNET.2005.860096",
    language = "English (US)",
    volume = "13",
    pages = "1219--1232",
    journal = "IEEE/ACM Transactions on Networking",
    issn = "1063-6692",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",
    number = "6",

    }

    TY - JOUR

    T1 - What's new

    T2 - Finding significant differences in network data streams

    AU - Cormode, Graham

    AU - Muthukrishnan, Shanmugavelayutham

    PY - 2005/12/1

    Y1 - 2005/12/1

    N2 - Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about "what's new." Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the mo st significant deltoids in high-speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.

    AB - Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about "what's new." Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the mo st significant deltoids in high-speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.

    KW - Change detection

    KW - Data streams

    KW - Deltoids

    KW - Network data analysis

    UR - http://www.scopus.com/inward/record.url?scp=29844455221&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=29844455221&partnerID=8YFLogxK

    U2 - 10.1109/TNET.2005.860096

    DO - 10.1109/TNET.2005.860096

    M3 - Article

    AN - SCOPUS:29844455221

    VL - 13

    SP - 1219

    EP - 1232

    JO - IEEE/ACM Transactions on Networking

    JF - IEEE/ACM Transactions on Networking

    SN - 1063-6692

    IS - 6

    ER -