What's new: Finding significant differences in network data streams

Graham Cormode, Shanmugavelayutham Muthukrishnan

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about "what's new". Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the most significant deltoids in high speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.

    Original languageEnglish (US)
    Title of host publicationIEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies
    Pages1534-1545
    Number of pages12
    DOIs
    StatePublished - Nov 22 2004
    EventIEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies - Hongkong, China
    Duration: Mar 7 2004Mar 11 2004

    Publication series

    NameProceedings - IEEE INFOCOM
    Volume3
    ISSN (Print)0743-166X

    Other

    OtherIEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies
    CountryChina
    CityHongkong
    Period3/7/043/11/04

    Fingerprint

    Routers
    Managers
    Monitoring

    Keywords

    • Network measurements
    • Traffic data analysis

    ASJC Scopus subject areas

    • Computer Science(all)
    • Electrical and Electronic Engineering

    Cite this

    Cormode, G., & Muthukrishnan, S. (2004). What's new: Finding significant differences in network data streams. In IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies (pp. 1534-1545). (Proceedings - IEEE INFOCOM; Vol. 3). https://doi.org/10.1109/INFCOM.2004.1354567

    What's new : Finding significant differences in network data streams. / Cormode, Graham; Muthukrishnan, Shanmugavelayutham.

    IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies. 2004. p. 1534-1545 (Proceedings - IEEE INFOCOM; Vol. 3).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Cormode, G & Muthukrishnan, S 2004, What's new: Finding significant differences in network data streams. in IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings - IEEE INFOCOM, vol. 3, pp. 1534-1545, IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies, Hongkong, China, 3/7/04. https://doi.org/10.1109/INFCOM.2004.1354567
    Cormode G, Muthukrishnan S. What's new: Finding significant differences in network data streams. In IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies. 2004. p. 1534-1545. (Proceedings - IEEE INFOCOM). https://doi.org/10.1109/INFCOM.2004.1354567
    Cormode, Graham ; Muthukrishnan, Shanmugavelayutham. / What's new : Finding significant differences in network data streams. IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies. 2004. pp. 1534-1545 (Proceedings - IEEE INFOCOM).
    @inproceedings{76553561875345fd8dd2e6d6b5652e2f,
    title = "What's new: Finding significant differences in network data streams",
    abstract = "Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about {"}what's new{"}. Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the most significant deltoids in high speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.",
    keywords = "Network measurements, Traffic data analysis",
    author = "Graham Cormode and Shanmugavelayutham Muthukrishnan",
    year = "2004",
    month = "11",
    day = "22",
    doi = "10.1109/INFCOM.2004.1354567",
    language = "English (US)",
    isbn = "0780383559",
    series = "Proceedings - IEEE INFOCOM",
    pages = "1534--1545",
    booktitle = "IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies",

    }

    TY - GEN

    T1 - What's new

    T2 - Finding significant differences in network data streams

    AU - Cormode, Graham

    AU - Muthukrishnan, Shanmugavelayutham

    PY - 2004/11/22

    Y1 - 2004/11/22

    N2 - Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about "what's new". Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the most significant deltoids in high speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.

    AB - Monitoring and analyzing network traffic usage patterns is vital for managing IP Networks. An important problem is to provide network managers with information about changes in traffic, informing them about "what's new". Specifically, we focus on the challenge of finding significantly large differences in traffic: over time, between interfaces and between routers. We introduce the idea of a deltoid: an item that has a large difference, whether the difference is absolute, relative or variational. We present novel algorithms for finding the most significant deltoids in high speed traffic data, and prove that they use small space, very small time per update, and are guaranteed to find significant deltoids with pre-specified accuracy. In experimental evaluation with real network traffic, our algorithms perform well and recover almost all deltoids. This is the first work to provide solutions capable of working over the data with one pass, at network traffic speeds.

    KW - Network measurements

    KW - Traffic data analysis

    UR - http://www.scopus.com/inward/record.url?scp=8344272783&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=8344272783&partnerID=8YFLogxK

    U2 - 10.1109/INFCOM.2004.1354567

    DO - 10.1109/INFCOM.2004.1354567

    M3 - Conference contribution

    SN - 0780383559

    T3 - Proceedings - IEEE INFOCOM

    SP - 1534

    EP - 1545

    BT - IEEE INFOCOM 2004 - Conference on Computer Communications - Twenty-Third Annual Joint Conference of the IEEE Computer and Communications Societies

    ER -