From a calculus to an execution environment for stream processing

Robert Soulé, Martin Hirzel, Buǧra Gedik, Robert Grimm

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

At one level, this paper is about River, a virtual execution environment for stream processing. Stream processing is a paradigm well-suited for many modern data processing systems that ingest high-volume data streams from the real world, such as audio/video streaming, high-frequency trading, and security monitoring. One attractive property of stream processing is that it lends itself to parallelization on multicores, and even to distribution on clusters when extreme scale is required. Stream processing has been co-evolved by several communities, leading to diverse languages with similar core concepts. Providing a common execution environment reduces language development effort and increases portability. We designed River as a practical realization of Brooklet, a calculus for stream processing. So at another level, this paper is about a journey from theory (the calculus) to practice (the execution environment). The challenge is that, by definition, a calculus abstracts away all but the most central concepts. Hence, there are several research questions in concretizing the missing parts, not to mention a significant engineering effort in implementing them. But the effort is well worth it, because using a calculus as a foundation yields clear semantics and proven correctness results.

Original languageEnglish (US)
Title of host publicationProceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12
Pages20-31
Number of pages12
DOIs
StatePublished - 2012
Event6th ACM International Conference on Distributed Event-Based Systems, DEBS'12 - Berlin, Germany
Duration: Jul 16 2012Jul 20 2012

Other

Other6th ACM International Conference on Distributed Event-Based Systems, DEBS'12
CountryGermany
CityBerlin
Period7/16/127/20/12

Fingerprint

Processing
Rivers
Video streaming
Virtual reality
Semantics
Monitoring

Keywords

  • CQL
  • Domain specific language
  • Intermediate language
  • Sawzall
  • Stream processing
  • StreamIt

ASJC Scopus subject areas

  • Computational Theory and Mathematics

Cite this

Soulé, R., Hirzel, M., Gedik, B., & Grimm, R. (2012). From a calculus to an execution environment for stream processing. In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12 (pp. 20-31) https://doi.org/10.1145/2335484.2335487

From a calculus to an execution environment for stream processing. / Soulé, Robert; Hirzel, Martin; Gedik, Buǧra; Grimm, Robert.

Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12. 2012. p. 20-31.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Soulé, R, Hirzel, M, Gedik, B & Grimm, R 2012, From a calculus to an execution environment for stream processing. in Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12. pp. 20-31, 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12, Berlin, Germany, 7/16/12. https://doi.org/10.1145/2335484.2335487
Soulé R, Hirzel M, Gedik B, Grimm R. From a calculus to an execution environment for stream processing. In Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12. 2012. p. 20-31 https://doi.org/10.1145/2335484.2335487
Soulé, Robert ; Hirzel, Martin ; Gedik, Buǧra ; Grimm, Robert. / From a calculus to an execution environment for stream processing. Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12. 2012. pp. 20-31
@inproceedings{0ce1572523794dbd81fd344150864b7c,
title = "From a calculus to an execution environment for stream processing",
abstract = "At one level, this paper is about River, a virtual execution environment for stream processing. Stream processing is a paradigm well-suited for many modern data processing systems that ingest high-volume data streams from the real world, such as audio/video streaming, high-frequency trading, and security monitoring. One attractive property of stream processing is that it lends itself to parallelization on multicores, and even to distribution on clusters when extreme scale is required. Stream processing has been co-evolved by several communities, leading to diverse languages with similar core concepts. Providing a common execution environment reduces language development effort and increases portability. We designed River as a practical realization of Brooklet, a calculus for stream processing. So at another level, this paper is about a journey from theory (the calculus) to practice (the execution environment). The challenge is that, by definition, a calculus abstracts away all but the most central concepts. Hence, there are several research questions in concretizing the missing parts, not to mention a significant engineering effort in implementing them. But the effort is well worth it, because using a calculus as a foundation yields clear semantics and proven correctness results.",
keywords = "CQL, Domain specific language, Intermediate language, Sawzall, Stream processing, StreamIt",
author = "Robert Soul{\'e} and Martin Hirzel and Buǧra Gedik and Robert Grimm",
year = "2012",
doi = "10.1145/2335484.2335487",
language = "English (US)",
isbn = "9781450313155",
pages = "20--31",
booktitle = "Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12",

}

TY - GEN

T1 - From a calculus to an execution environment for stream processing

AU - Soulé, Robert

AU - Hirzel, Martin

AU - Gedik, Buǧra

AU - Grimm, Robert

PY - 2012

Y1 - 2012

N2 - At one level, this paper is about River, a virtual execution environment for stream processing. Stream processing is a paradigm well-suited for many modern data processing systems that ingest high-volume data streams from the real world, such as audio/video streaming, high-frequency trading, and security monitoring. One attractive property of stream processing is that it lends itself to parallelization on multicores, and even to distribution on clusters when extreme scale is required. Stream processing has been co-evolved by several communities, leading to diverse languages with similar core concepts. Providing a common execution environment reduces language development effort and increases portability. We designed River as a practical realization of Brooklet, a calculus for stream processing. So at another level, this paper is about a journey from theory (the calculus) to practice (the execution environment). The challenge is that, by definition, a calculus abstracts away all but the most central concepts. Hence, there are several research questions in concretizing the missing parts, not to mention a significant engineering effort in implementing them. But the effort is well worth it, because using a calculus as a foundation yields clear semantics and proven correctness results.

AB - At one level, this paper is about River, a virtual execution environment for stream processing. Stream processing is a paradigm well-suited for many modern data processing systems that ingest high-volume data streams from the real world, such as audio/video streaming, high-frequency trading, and security monitoring. One attractive property of stream processing is that it lends itself to parallelization on multicores, and even to distribution on clusters when extreme scale is required. Stream processing has been co-evolved by several communities, leading to diverse languages with similar core concepts. Providing a common execution environment reduces language development effort and increases portability. We designed River as a practical realization of Brooklet, a calculus for stream processing. So at another level, this paper is about a journey from theory (the calculus) to practice (the execution environment). The challenge is that, by definition, a calculus abstracts away all but the most central concepts. Hence, there are several research questions in concretizing the missing parts, not to mention a significant engineering effort in implementing them. But the effort is well worth it, because using a calculus as a foundation yields clear semantics and proven correctness results.

KW - CQL

KW - Domain specific language

KW - Intermediate language

KW - Sawzall

KW - Stream processing

KW - StreamIt

UR - http://www.scopus.com/inward/record.url?scp=84865046741&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84865046741&partnerID=8YFLogxK

U2 - 10.1145/2335484.2335487

DO - 10.1145/2335484.2335487

M3 - Conference contribution

SN - 9781450313155

SP - 20

EP - 31

BT - Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12

ER -