A heartbeat mechanism and its application in gigascope

Theodore Johnson, Shanmugavelayutham Muthukrishnan, Vladislav Shkapenyuk, Oliver Spatscheck

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application. In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.

    Original languageEnglish (US)
    Title of host publicationVLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases
    Pages1079-1088
    Number of pages10
    StatePublished - Dec 1 2005
    EventVLDB 2005 - 31st International Conference on Very Large Data Bases - Trondheim, Norway
    Duration: Aug 30 2005Sep 2 2005

    Publication series

    NameVLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases
    Volume3

    Other

    OtherVLDB 2005 - 31st International Conference on Very Large Data Bases
    CountryNorway
    CityTrondheim
    Period8/30/059/2/05

    Fingerprint

    Monitoring
    Industrial applications
    Data storage equipment

    ASJC Scopus subject areas

    • Engineering(all)

    Cite this

    Johnson, T., Muthukrishnan, S., Shkapenyuk, V., & Spatscheck, O. (2005). A heartbeat mechanism and its application in gigascope. In VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases (pp. 1079-1088). (VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases; Vol. 3).

    A heartbeat mechanism and its application in gigascope. / Johnson, Theodore; Muthukrishnan, Shanmugavelayutham; Shkapenyuk, Vladislav; Spatscheck, Oliver.

    VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases. 2005. p. 1079-1088 (VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases; Vol. 3).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Johnson, T, Muthukrishnan, S, Shkapenyuk, V & Spatscheck, O 2005, A heartbeat mechanism and its application in gigascope. in VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases. VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases, vol. 3, pp. 1079-1088, VLDB 2005 - 31st International Conference on Very Large Data Bases, Trondheim, Norway, 8/30/05.
    Johnson T, Muthukrishnan S, Shkapenyuk V, Spatscheck O. A heartbeat mechanism and its application in gigascope. In VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases. 2005. p. 1079-1088. (VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases).
    Johnson, Theodore ; Muthukrishnan, Shanmugavelayutham ; Shkapenyuk, Vladislav ; Spatscheck, Oliver. / A heartbeat mechanism and its application in gigascope. VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases. 2005. pp. 1079-1088 (VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases).
    @inproceedings{e8add51a78ba4b6d9429a9ed9e0de727,
    title = "A heartbeat mechanism and its application in gigascope",
    abstract = "Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application. In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.",
    author = "Theodore Johnson and Shanmugavelayutham Muthukrishnan and Vladislav Shkapenyuk and Oliver Spatscheck",
    year = "2005",
    month = "12",
    day = "1",
    language = "English (US)",
    isbn = "1595931546",
    series = "VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases",
    pages = "1079--1088",
    booktitle = "VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases",

    }

    TY - GEN

    T1 - A heartbeat mechanism and its application in gigascope

    AU - Johnson, Theodore

    AU - Muthukrishnan, Shanmugavelayutham

    AU - Shkapenyuk, Vladislav

    AU - Spatscheck, Oliver

    PY - 2005/12/1

    Y1 - 2005/12/1

    N2 - Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application. In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.

    AB - Data stream management systems often rely on ordering properties of tuple attributes in order to implement non-blocking operators. However, query operators that work with multiple streams, such as stream merge or join, can often still block if one of the input stream is very slow or bursty. In principle, punctuation and heartbeat mechanisms have been proposed to unblock streaming operators. In practice, it is a challenge to incorporate such mechanisms into a high-performance stream management system that is operational in an industrial application. In this paper, we introduce a system for punctuation-carrying heartbeat generation that we developed for Gigascope, a high-performance streaming database for network monitoring, that is operationally used within AT&T's IP backbone. We show how heartbeats can be regularly generated by low-level nodes in query execution plans and propagated upward unblocking all streaming operators on its way. Additionally, our heartbeat mechanism can be used for other applications in distributed settings such as detecting node failures, performance monitoring, and query optimization. A performance evaluation using live data feeds shows that our system is capable of working at multiple Gigabit line speeds in a live, industrial deployment and can significantly decrease the query memory utilization.

    UR - http://www.scopus.com/inward/record.url?scp=33745629862&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=33745629862&partnerID=8YFLogxK

    M3 - Conference contribution

    AN - SCOPUS:33745629862

    SN - 1595931546

    SN - 9781595931542

    T3 - VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases

    SP - 1079

    EP - 1088

    BT - VLDB 2005 - Proceedings of 31st International Conference on Very Large Data Bases

    ER -