Protocol design contests

Anirudh Sivaraman, Keith Winstein, Pauline Varley, João Batalha, Ameesh Goyal, Somak Das, Joshua Ma, Hari Balakrishnan

Research output: Contribution to journalArticle

Abstract

In fields like data mining and natural language processing, design contests have been successfully used to advance the state of the art. Such contests offer an opportunity to bring the excitement and challenges of protocol design-one of the core intellectual elements of research and practice in networked systems-to a broader group of potential contributors, whose ideas may prove important. Moreover, it may lead to an increase in the number of students, especially undergraduates or those learning via online courses, interested in pursuing a career in the field. We describe the creation of the infrastructure and our experience with a protocol design contest conducted in MIT's graduate Computer Networks class. This contest involved the design and evaluation of a congestion-control protocol for paths traversing cellular wireless networks. One key to the success of a design contest is an unambiguous, measurable objective to compare protocols. In practice, protocol design is the art of trading off conflicting goals with each other, but in this contest, we specified that the goal was to maximize log(throughput=delay). This goal is a good match for applications such as video streaming or videoconferencing that care about high throughput and low interactive delays. Some students produced protocols whose performance was better than published protocols tackling similar goals. Furthermore, the convex hull of the set of all student protocols traced out a tradeoff curve in the throughput-delay space, providing useful insights into the entire space of possible protocols. We found that student protocols diverged in performance between the training and testing traces, indicating that some students had overtrained ("overfitted") their protocols to the training trace. Our conclusion is that, if designed properly, such contests could benefit networking research by making new proposals more easily reproducible and amenable to such "gamification," improve networked systems, and provide an avenue for outreach.

Original languageEnglish (US)
Pages (from-to)38-44
Number of pages7
JournalComputer Communication Review
Volume44
Issue number3
DOIs
StatePublished - 2014

Fingerprint

Network protocols
Students
Throughput
Video streaming
Computer networks
Data mining
Wireless networks
Testing
Processing

Keywords

  • Congestion control
  • Design contest
  • Gamification
  • Machine learning
  • Protocol

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications

Cite this

Sivaraman, A., Winstein, K., Varley, P., Batalha, J., Goyal, A., Das, S., ... Balakrishnan, H. (2014). Protocol design contests. Computer Communication Review, 44(3), 38-44. https://doi.org/10.1145/2656877.2656883

Protocol design contests. / Sivaraman, Anirudh; Winstein, Keith; Varley, Pauline; Batalha, João; Goyal, Ameesh; Das, Somak; Ma, Joshua; Balakrishnan, Hari.

In: Computer Communication Review, Vol. 44, No. 3, 2014, p. 38-44.

Research output: Contribution to journalArticle

Sivaraman, A, Winstein, K, Varley, P, Batalha, J, Goyal, A, Das, S, Ma, J & Balakrishnan, H 2014, 'Protocol design contests', Computer Communication Review, vol. 44, no. 3, pp. 38-44. https://doi.org/10.1145/2656877.2656883
Sivaraman A, Winstein K, Varley P, Batalha J, Goyal A, Das S et al. Protocol design contests. Computer Communication Review. 2014;44(3):38-44. https://doi.org/10.1145/2656877.2656883
Sivaraman, Anirudh ; Winstein, Keith ; Varley, Pauline ; Batalha, João ; Goyal, Ameesh ; Das, Somak ; Ma, Joshua ; Balakrishnan, Hari. / Protocol design contests. In: Computer Communication Review. 2014 ; Vol. 44, No. 3. pp. 38-44.
@article{20c37dfd2dd74544ae559180a9107b5b,
title = "Protocol design contests",
abstract = "In fields like data mining and natural language processing, design contests have been successfully used to advance the state of the art. Such contests offer an opportunity to bring the excitement and challenges of protocol design-one of the core intellectual elements of research and practice in networked systems-to a broader group of potential contributors, whose ideas may prove important. Moreover, it may lead to an increase in the number of students, especially undergraduates or those learning via online courses, interested in pursuing a career in the field. We describe the creation of the infrastructure and our experience with a protocol design contest conducted in MIT's graduate Computer Networks class. This contest involved the design and evaluation of a congestion-control protocol for paths traversing cellular wireless networks. One key to the success of a design contest is an unambiguous, measurable objective to compare protocols. In practice, protocol design is the art of trading off conflicting goals with each other, but in this contest, we specified that the goal was to maximize log(throughput=delay). This goal is a good match for applications such as video streaming or videoconferencing that care about high throughput and low interactive delays. Some students produced protocols whose performance was better than published protocols tackling similar goals. Furthermore, the convex hull of the set of all student protocols traced out a tradeoff curve in the throughput-delay space, providing useful insights into the entire space of possible protocols. We found that student protocols diverged in performance between the training and testing traces, indicating that some students had overtrained ({"}overfitted{"}) their protocols to the training trace. Our conclusion is that, if designed properly, such contests could benefit networking research by making new proposals more easily reproducible and amenable to such {"}gamification,{"} improve networked systems, and provide an avenue for outreach.",
keywords = "Congestion control, Design contest, Gamification, Machine learning, Protocol",
author = "Anirudh Sivaraman and Keith Winstein and Pauline Varley and Jo{\~a}o Batalha and Ameesh Goyal and Somak Das and Joshua Ma and Hari Balakrishnan",
year = "2014",
doi = "10.1145/2656877.2656883",
language = "English (US)",
volume = "44",
pages = "38--44",
journal = "Computer Communication Review",
issn = "0146-4833",
publisher = "Association for Computing Machinery (ACM)",
number = "3",

}

TY - JOUR

T1 - Protocol design contests

AU - Sivaraman, Anirudh

AU - Winstein, Keith

AU - Varley, Pauline

AU - Batalha, João

AU - Goyal, Ameesh

AU - Das, Somak

AU - Ma, Joshua

AU - Balakrishnan, Hari

PY - 2014

Y1 - 2014

N2 - In fields like data mining and natural language processing, design contests have been successfully used to advance the state of the art. Such contests offer an opportunity to bring the excitement and challenges of protocol design-one of the core intellectual elements of research and practice in networked systems-to a broader group of potential contributors, whose ideas may prove important. Moreover, it may lead to an increase in the number of students, especially undergraduates or those learning via online courses, interested in pursuing a career in the field. We describe the creation of the infrastructure and our experience with a protocol design contest conducted in MIT's graduate Computer Networks class. This contest involved the design and evaluation of a congestion-control protocol for paths traversing cellular wireless networks. One key to the success of a design contest is an unambiguous, measurable objective to compare protocols. In practice, protocol design is the art of trading off conflicting goals with each other, but in this contest, we specified that the goal was to maximize log(throughput=delay). This goal is a good match for applications such as video streaming or videoconferencing that care about high throughput and low interactive delays. Some students produced protocols whose performance was better than published protocols tackling similar goals. Furthermore, the convex hull of the set of all student protocols traced out a tradeoff curve in the throughput-delay space, providing useful insights into the entire space of possible protocols. We found that student protocols diverged in performance between the training and testing traces, indicating that some students had overtrained ("overfitted") their protocols to the training trace. Our conclusion is that, if designed properly, such contests could benefit networking research by making new proposals more easily reproducible and amenable to such "gamification," improve networked systems, and provide an avenue for outreach.

AB - In fields like data mining and natural language processing, design contests have been successfully used to advance the state of the art. Such contests offer an opportunity to bring the excitement and challenges of protocol design-one of the core intellectual elements of research and practice in networked systems-to a broader group of potential contributors, whose ideas may prove important. Moreover, it may lead to an increase in the number of students, especially undergraduates or those learning via online courses, interested in pursuing a career in the field. We describe the creation of the infrastructure and our experience with a protocol design contest conducted in MIT's graduate Computer Networks class. This contest involved the design and evaluation of a congestion-control protocol for paths traversing cellular wireless networks. One key to the success of a design contest is an unambiguous, measurable objective to compare protocols. In practice, protocol design is the art of trading off conflicting goals with each other, but in this contest, we specified that the goal was to maximize log(throughput=delay). This goal is a good match for applications such as video streaming or videoconferencing that care about high throughput and low interactive delays. Some students produced protocols whose performance was better than published protocols tackling similar goals. Furthermore, the convex hull of the set of all student protocols traced out a tradeoff curve in the throughput-delay space, providing useful insights into the entire space of possible protocols. We found that student protocols diverged in performance between the training and testing traces, indicating that some students had overtrained ("overfitted") their protocols to the training trace. Our conclusion is that, if designed properly, such contests could benefit networking research by making new proposals more easily reproducible and amenable to such "gamification," improve networked systems, and provide an avenue for outreach.

KW - Congestion control

KW - Design contest

KW - Gamification

KW - Machine learning

KW - Protocol

UR - http://www.scopus.com/inward/record.url?scp=84905820659&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84905820659&partnerID=8YFLogxK

U2 - 10.1145/2656877.2656883

DO - 10.1145/2656877.2656883

M3 - Article

VL - 44

SP - 38

EP - 44

JO - Computer Communication Review

JF - Computer Communication Review

SN - 0146-4833

IS - 3

ER -