TCP bbr for ultra-low latency networking: challenges, analysis, and solutions

Rajeev Kumar, Athanasios Koutsaftis, Fraida Fund, Gaurang Naik, Pei Liu, Yong Liu, Shivendra Panwar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the new emerging throughput-intensive ultralow latency applications, there is a need for a transport layer protocol that can achieve high throughput with low latency. One promising candidate is TCP BBR, a protocol developed by Google, with the aim of achieving high throughput and low latency by operating around the Bandwidth Delay Product (BDP) of the bottleneck link. Google reported significant throughput gains and much lower latency relative to TCP Cubic following the deployment of BBR in their high-speed wide area wired network. As most of these emerging applications will be supported by Millimeter Wave (mmWave) wireless networks, BBR should achieve both high throughput and ultra-low latency in these settings. However, in our preliminary experiments with BBR over a mmWave wireless link operating at 60 GHz, we observed a severe degradation in throughput that we were able to attribute to high delay variation on the link. In this paper, we show that 'throughput collapse' occurs when BBR's estimate of minimum RTT is less than half of the average RTT of the uncongested link (as when delay jitter is large). We demonstrate this phenomenon and explain the underlying reasons for it using a series of controlled experiments on the CloudLab testbed. We also present a mathematical analysis of BBR, which matches our experimental results closely. Based on our analysis, we propose and experimentally evaluate potential solutions that can overcome the throughput collapse without addina sianificant latency.

Original languageEnglish (US)
Title of host publication2019 IFIP Networking Conference, IFIP Networking 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9783903176164
DOIs
StatePublished - May 2019
Event2019 IFIP Networking Conference, IFIP Networking 2019 - Warsaw, Poland
Duration: May 20 2019May 22 2019

Publication series

Name2019 IFIP Networking Conference, IFIP Networking 2019

Conference

Conference2019 IFIP Networking Conference, IFIP Networking 2019
CountryPoland
CityWarsaw
Period5/20/195/22/19

Fingerprint

Throughput
Millimeter waves
Telecommunication links
Network protocols
HIgh speed networks
Wide area networks
Networking
Latency
Testbeds
Jitter
Wireless networks
Experiments
Bandwidth
Degradation

Keywords

  • 5G
  • BBR
  • Congestion Control
  • Millimeter Wave
  • RTT
  • TCP

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality

Cite this

Kumar, R., Koutsaftis, A., Fund, F., Naik, G., Liu, P., Liu, Y., & Panwar, S. (2019). TCP bbr for ultra-low latency networking: challenges, analysis, and solutions. In 2019 IFIP Networking Conference, IFIP Networking 2019 [8816856] (2019 IFIP Networking Conference, IFIP Networking 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.23919/IFIPNetworking.2019.8816856

TCP bbr for ultra-low latency networking : challenges, analysis, and solutions. / Kumar, Rajeev; Koutsaftis, Athanasios; Fund, Fraida; Naik, Gaurang; Liu, Pei; Liu, Yong; Panwar, Shivendra.

2019 IFIP Networking Conference, IFIP Networking 2019. Institute of Electrical and Electronics Engineers Inc., 2019. 8816856 (2019 IFIP Networking Conference, IFIP Networking 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Kumar, R, Koutsaftis, A, Fund, F, Naik, G, Liu, P, Liu, Y & Panwar, S 2019, TCP bbr for ultra-low latency networking: challenges, analysis, and solutions. in 2019 IFIP Networking Conference, IFIP Networking 2019., 8816856, 2019 IFIP Networking Conference, IFIP Networking 2019, Institute of Electrical and Electronics Engineers Inc., 2019 IFIP Networking Conference, IFIP Networking 2019, Warsaw, Poland, 5/20/19. https://doi.org/10.23919/IFIPNetworking.2019.8816856
Kumar R, Koutsaftis A, Fund F, Naik G, Liu P, Liu Y et al. TCP bbr for ultra-low latency networking: challenges, analysis, and solutions. In 2019 IFIP Networking Conference, IFIP Networking 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 8816856. (2019 IFIP Networking Conference, IFIP Networking 2019). https://doi.org/10.23919/IFIPNetworking.2019.8816856
Kumar, Rajeev ; Koutsaftis, Athanasios ; Fund, Fraida ; Naik, Gaurang ; Liu, Pei ; Liu, Yong ; Panwar, Shivendra. / TCP bbr for ultra-low latency networking : challenges, analysis, and solutions. 2019 IFIP Networking Conference, IFIP Networking 2019. Institute of Electrical and Electronics Engineers Inc., 2019. (2019 IFIP Networking Conference, IFIP Networking 2019).
@inproceedings{55701dbe18f0421584a2a9e6a2cf162e,
title = "TCP bbr for ultra-low latency networking: challenges, analysis, and solutions",
abstract = "With the new emerging throughput-intensive ultralow latency applications, there is a need for a transport layer protocol that can achieve high throughput with low latency. One promising candidate is TCP BBR, a protocol developed by Google, with the aim of achieving high throughput and low latency by operating around the Bandwidth Delay Product (BDP) of the bottleneck link. Google reported significant throughput gains and much lower latency relative to TCP Cubic following the deployment of BBR in their high-speed wide area wired network. As most of these emerging applications will be supported by Millimeter Wave (mmWave) wireless networks, BBR should achieve both high throughput and ultra-low latency in these settings. However, in our preliminary experiments with BBR over a mmWave wireless link operating at 60 GHz, we observed a severe degradation in throughput that we were able to attribute to high delay variation on the link. In this paper, we show that 'throughput collapse' occurs when BBR's estimate of minimum RTT is less than half of the average RTT of the uncongested link (as when delay jitter is large). We demonstrate this phenomenon and explain the underlying reasons for it using a series of controlled experiments on the CloudLab testbed. We also present a mathematical analysis of BBR, which matches our experimental results closely. Based on our analysis, we propose and experimentally evaluate potential solutions that can overcome the throughput collapse without addina sianificant latency.",
keywords = "5G, BBR, Congestion Control, Millimeter Wave, RTT, TCP",
author = "Rajeev Kumar and Athanasios Koutsaftis and Fraida Fund and Gaurang Naik and Pei Liu and Yong Liu and Shivendra Panwar",
year = "2019",
month = "5",
doi = "10.23919/IFIPNetworking.2019.8816856",
language = "English (US)",
series = "2019 IFIP Networking Conference, IFIP Networking 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "2019 IFIP Networking Conference, IFIP Networking 2019",

}

TY - GEN

T1 - TCP bbr for ultra-low latency networking

T2 - challenges, analysis, and solutions

AU - Kumar, Rajeev

AU - Koutsaftis, Athanasios

AU - Fund, Fraida

AU - Naik, Gaurang

AU - Liu, Pei

AU - Liu, Yong

AU - Panwar, Shivendra

PY - 2019/5

Y1 - 2019/5

N2 - With the new emerging throughput-intensive ultralow latency applications, there is a need for a transport layer protocol that can achieve high throughput with low latency. One promising candidate is TCP BBR, a protocol developed by Google, with the aim of achieving high throughput and low latency by operating around the Bandwidth Delay Product (BDP) of the bottleneck link. Google reported significant throughput gains and much lower latency relative to TCP Cubic following the deployment of BBR in their high-speed wide area wired network. As most of these emerging applications will be supported by Millimeter Wave (mmWave) wireless networks, BBR should achieve both high throughput and ultra-low latency in these settings. However, in our preliminary experiments with BBR over a mmWave wireless link operating at 60 GHz, we observed a severe degradation in throughput that we were able to attribute to high delay variation on the link. In this paper, we show that 'throughput collapse' occurs when BBR's estimate of minimum RTT is less than half of the average RTT of the uncongested link (as when delay jitter is large). We demonstrate this phenomenon and explain the underlying reasons for it using a series of controlled experiments on the CloudLab testbed. We also present a mathematical analysis of BBR, which matches our experimental results closely. Based on our analysis, we propose and experimentally evaluate potential solutions that can overcome the throughput collapse without addina sianificant latency.

AB - With the new emerging throughput-intensive ultralow latency applications, there is a need for a transport layer protocol that can achieve high throughput with low latency. One promising candidate is TCP BBR, a protocol developed by Google, with the aim of achieving high throughput and low latency by operating around the Bandwidth Delay Product (BDP) of the bottleneck link. Google reported significant throughput gains and much lower latency relative to TCP Cubic following the deployment of BBR in their high-speed wide area wired network. As most of these emerging applications will be supported by Millimeter Wave (mmWave) wireless networks, BBR should achieve both high throughput and ultra-low latency in these settings. However, in our preliminary experiments with BBR over a mmWave wireless link operating at 60 GHz, we observed a severe degradation in throughput that we were able to attribute to high delay variation on the link. In this paper, we show that 'throughput collapse' occurs when BBR's estimate of minimum RTT is less than half of the average RTT of the uncongested link (as when delay jitter is large). We demonstrate this phenomenon and explain the underlying reasons for it using a series of controlled experiments on the CloudLab testbed. We also present a mathematical analysis of BBR, which matches our experimental results closely. Based on our analysis, we propose and experimentally evaluate potential solutions that can overcome the throughput collapse without addina sianificant latency.

KW - 5G

KW - BBR

KW - Congestion Control

KW - Millimeter Wave

KW - RTT

KW - TCP

UR - http://www.scopus.com/inward/record.url?scp=85072792959&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072792959&partnerID=8YFLogxK

U2 - 10.23919/IFIPNetworking.2019.8816856

DO - 10.23919/IFIPNetworking.2019.8816856

M3 - Conference contribution

AN - SCOPUS:85072792959

T3 - 2019 IFIP Networking Conference, IFIP Networking 2019

BT - 2019 IFIP Networking Conference, IFIP Networking 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -