Multipath TCP APIs

Shortly after the publication of RFC 6824, the IETF published RFC 6827. This RFC provides a first discussion of how a socket interface could be designed to support Multipath TCP. When RFC 6827 was published, none of the existing implementations included a socket API for Multipath TCP. The Linux implementation used the standard sockets with a few options and sysctls. This enabled Multipath TCP to be useable by any application, but without special means to control the utilisation of the subflows. This ability to support standard application remains one of the main advantages of this implementation. Once Multipath TCP has been added in the Linux kernel, all applications immediately benefit from its multipath capabilities.

However, there are some used cases where the applications know that they are using Multipath TCP and want to control some of its features such as the utilisation of the subflows of the packet scheduler. A first approach to provide this control was proposed in [HDB+15]. This solution exposes the Multipath TCP kernel API to user space applications through netlink messages.

../../../_images/netlink.png

Thanks to these messages, it is possible to design a userspace path manager librayr that control the utilisation of the subflows. This is important for use cases where one path is more costly than the other, e.g. on smartphones where the celullar interface is billed on a volume basis while the Wi-Fi interface is not metered. Several use cases and examples are described in [HDB+15].

A second possible approach is to extend the traditional socket API by using socket options that can be set or retrieved by the applications [HB16]. Six new socket options are proposed in this article and described later in draft-hesmans-mptcp-socket-03.

../../../_images/mptcp-socket.png

These socket options enable an application to control the creation and the removal of subflows during the lifetime of a Multipath TCP connection. Two use cases are described in this short paper : refreshing subflows on a regular basis and delaying the establishment of subflows. The patches that implement these socket options have been distributed on the mptcp-dev mailing list but not yet integrated in the official releases of the Multipath TCP implementation in the Linux kernel.

Apple has also defined a specific API to enable applications to interact with Multipath TCP. A simple example was provided in [HB16].

../../../_images/socket-apple.png

This API was private and not officially supported by Apple although there were examples in the Apple source code. When Apple announced Multipath TCP support for all iOS11 application, they also released a simple API that extends the URLSessionConfiguration property. This API does not provide a fine control to the utilization of the subflows as the socket API implemented on Linux, but it allows applications to select between the handover and interactive modes. Additional information is available from https://developer.apple.com/documentation/foundation/urlsessionconfiguration/improving_network_reliability_using_multipath_tcp

../../../_images/apple-url.png

References

[HB16](1, 2) Benjamin Hesmans and Olivier Bonaventure. An enhanced socket api for multipath tcp. In Proceeding ANRW ‘16 Proceedings of the 2016 Applied Networking Research Workshop. 2016. URL: https://inl.info.ucl.ac.be/publications/enhanced-socket-api-multipath-tcp.html.
[HDB+15](1, 2) Benjamin Hesmans, Gregory Detal, Sebastien Barre, Raphael Bauduin, and Olivier Bonaventure. Smapp : towards smart multipath tcp-enabled applications. In Proc. Conext 2015. Heidelberg, December 2015. URL: https://inl.info.ucl.ac.be/publications/smapp-towards-smart-multipath-tcp-enabled-applications.html.

Evolution of the documents produced by the MPTCP working group

Since its creation, the mptcp working group of the IETF has produced 7 documents that were published as RFCs and an eighth one is currently in last call. This post provides a brief description of these different documents as a starting point for someone who wants to start to look at multipath transport protocols.

../../../_images/rfcs.png

The working group has produced two Experimental RFCs and five Informational ones. The first two RFCs are RFC 6181 which discusses the security issues that were taken into account for the design of Multipath TCP and RFC 6182 which describes the basic architectural principles for Multipath TCP. RFC 6182 has already been discussed in a previous blog post. RFC 6181 was written by Marcelo Bagnulo and builds upon earlier work on a security analysis of Mobile IPv6 in RFC 4225 and shim6 RFC 4218. The key security issues that are discussed in this document are :

  • a flooding attack where an attacker who forces a server to send packets to a victim address by trying to force the server to use a subflow towards the victim. This attack was a concern for network layer protocols, but for Multipath TCP, this is not an issue since Multipath TCP validates the creation of the subflows with a three-way handshake and the MP_JOIN option.
  • a hijacking attack where an attacker leverages the address agility mechanisms of Multipath TCP to hijack an existing connection. One concern was the risk of an attacker being able to create a man-in-the-middle attack against existing Multipath TCP. This attack is prevented by the mechanism used by Multipath TCP to create subflows.
  • a discussion of time-shifted hijacking attacks

The threats analysis continued ad was later expanded in RFC 7430. This RFC introduces other types of attacks : an attack on ADD_ADDR which could allow an off-path attacker to create man-in-the-middle attack but under very unlikely circumstances, a denial of service attack on MP_JOIN, a SYN flood amplification attack and an eavesdropper that observes the initial handshake. These attacks were considered in the design of Multipath TCP or their implementations.

The other Informational RFCs are RFC 6897 which discusses API considerations and RFC 8041 which describes the known use cases where Multipath TCP has been deployed. Some of these use cases are also discussed in [BS16].

The three main documents produced by the mptcp working group are RFC 6356 which defines the coupled congestion control scheme and RFC 6824. As their publication date suggests, the congestion control scheme was stable much earlier and the protocol specification.

Experimental and Standards-track RFCs contain MUST, SHOULD and other keywords defined in RFC 2119. As RFC 6356 specifies a congestion control scheme, it only include two MUST keywords. However, RFC 6824 provides a precise protocol specification that leverages these keywords.

../../../_images/draft-ietf-mptcp-multiaddressed_keywords.png

The above figure, plotted with a script developed by Maxime Piraux for [PDCB18], shows the evolution of the utilization of the RFC 2119 keywords in the different versions of draft-ietf-mptcp-multiaddressed. There were three main phases in the utilization of these keywords: a low usage in the first four drafts, then a sudden increase in draft-ietf-mptcp-multiaddressed-03 that included changes in the MP_CAPABLE option, the addition of an address identifier in MP_PRIO and answers to review comments a huge list of changes.

It is interesting to observe the evolution of draft-ietf-mptcp-rfc6824bis

../../../_images/draft-ietf-mptcp-rfc6824bis_keywords.png

The above figure shows that when looking at the RFC 2119 keywords, the specification did not change a lot compared to RFC 6824. Most of the changes were clarifications except for the redefinition of the MP_CAPABLE option as discussed in a previous blog post.

In contrast, the transport part of QUIC, defined in draft-ietf-quic-transport appears to be much more complex when counting the RFC 2119 keywords and the specification is not yet finished.

../../../_images/draft-ietf-quic-transport_keywords.png

References

[BS16]Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/.
[PDCB18]Maxime Piraux, Quentin De Coninck, and Olivier Bonaventure. Observing the evolution of quic implementations. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC, EPIQ‘18, 8–14. New York, NY, USA, 2018. ACM. URL: https://quic-tracker.info.ucl.ac.be/blog/results/paper/2018/11/19/epiq-18-paper-accepted.html, doi:10.1145/3284850.3284852.

Apple uses Multipath TCP

The initial specification for Multipath TCP was published in January 2013 RFC 6824. Apple had participated to some of the discussions during the IETF meetings before, but never announced a deployment. Shortly after the publication of RFC 6824, Phil Eardley published a blank internet draft, draft-eardley-mptcp-implementations-survey-01 that explicitly asked questions to implementers.

../../../_images/survey.png

Four implementations were disclosed during the summer of 2013:

  • the stable Linux implementation discussed on page 11
  • an ongoing implementation on FreeBSD discussed on page 19
  • an anonymous implementation discussed on page 23
  • on implementation on Citrix load balancers discussed on page 31

Five years after, it is interesting to look at the characteristics of this anonymous implementation.

  • This implementation only supports client-initiated subflows
  • It uses 4 bytes DSN as a default, but can support 8 bytes DSN
  • The support for ADD_ADDR and REMOVE_ADDR was described as : It does not support sending ADD_ADDR or processing ADD_ADDR as it is considered a security risk. Also, we only have a client side implementation at the moment which always initiates the sub flows. The remote end does not send ADD_ADDR in our configuration. The client can send REMOVE_ADDR however when one of the established sub flow’s source address goes away. The client ignores incoming REMOVE_ADDR options also.
  • It does not implement the coupled congestion control defined in RFC 6356
  • It uses a private API and not the socket API proposed in RFC 6897
  • The proposed deployment is described as follows : MPTCP in mobile environments is very powerful when used in the active/backup mode. Since the network interfaces available on mobile devices have different cost characteristics as well as different bring up and power usage characteristics, it is not useful to share load across all available network interfaces - at least not currently. Providing session continuity across changing network environments is the key deployment scenario.

In September 2013, Apple launched iOS7 that included support for Multipath TCP. Apple’s motivation for using Multipath TCP on iOS have been explained in details in [BS16]:

Siri is the digital assistant in Apple’s iOS and macOS operating systems. Because speech recognition requires tremendous processing power, Siri streams spoken commands to Apple’s datacenter for speech recognition; the result is sent back to the smartphone. Although the duration of a user’s interaction with Siri is relatively short, Siri’s usage pattern made this data transfer a perfect client for MPTCP.

Many people use Siri while walking or driving. As they move farther away from a WiFi access point, the TCP connection used by Siri to stream its voice eventually fails, resulting in error messages.

To address this issue, Apple has been using MPTCP—and benefiting from its handover capabilities—since its iOS 7 release. When a user issues a Siri voice command, iOS establishes an MPTCP connection over WiFi and cellular. If the phone loses connectivity to the WiFi access point, traffic is handed over to the cellular interface. A WiFi connection that is still in sight of an access point can have a channel become so lossy that barely any segments can be transmitted. In this case, another retransmission timeout happens and iOS retransmits the traffic over the cellular link.

The article continues with additional information that describes how Apple has tuned Multipath TCP to this specific use case. A description of the Multipath TCP handshake used by Siri has been published in a previous blog post.

While Multipath TCP was part of iOS, it was only used by Apple’s own Siri applications. The regular applications could not leverage the benefits of Multipath TCP. This changed in 2017 with the launch of iOS11. During WDC2017, Christoph Paasch and his colleagues announced that any application would be able to use Multipath TCP on iOS11.

../../../_images/christoph.png

A detailed summary of these announcements appear on the tessares blog. iOS11 supports two modes of operation : Handover and Interactive.

Connection starts over the WiFi link and no packet is sent over the cellular interface. If the signal gets worse, a new TCP subflow will be created on the cellular interface automatically. The cellular subflow will be removed once the user is back in a WiFi network.

../../../_images/handover-apple.png

The interactive mode establishes both WiFi and cellular subflows for each Multipath TCP connection, even if the WiFi network appears to be working well. The objective of this mode is to reduce latency. The Multipath TCP scheduler will select the flow that provides the lowest latency.

../../../_images/interactive.png

Since the publication of iOS11, some applications have started to use Multipath TCP. One of them is the Multipath Tester, an application written by Quentin De Coninck that allows to compare the performance of Multipath TCP and Multipath QUIC on iOS11 [CB18]. You can download it from https://itunes.apple.com/us/app/multipathtester/id1351286809

../../../_images/mptester.png

References

[BS16]Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/.
[CB18]Quentin De Coninck and Olivier Bonaventure. Observing network handovers with multipath TCP. In Proceedings of the ACM SIGCOMM 2018 Conference on Posters and Demos, SIGCOMM 2018, Budapest, Hungary, August 20-25, 2018, 54–56. 2018. URL: https://multipath-quic.org/multipathtester/2018/08/28/sigcomm-poster.html, doi:10.1145/3234200.3234214.

Can Multipath TCP cope with middleboxes ?

As explained in a previous blog post, Multipath TCP had to cope with a variety of middleboxes which could interfere with this TCP extension.

Shortly after we detected the first interferences between a firewall and Multipath TCP, Honda et al. presented a detailed analysis [HNR+11] of the limits of the extensibility of TCP based on Internet measurements. To correctly understand the problems caused by middleboxes, we first need to remember that they can operate in any layer of the protocol stack as illustrated in the figure below.

../../../_images/mbox1.png

When a router forwards an IPv4 packet that contains a TCP segment, it may modify some fields of the IPv4 header but never changes any field of the TCP header. This is one of the basis of the layering principles.

../../../_images/mbox2.png

Middleboxes are different. As they potentially operate in any layer of the protocol stack, they can potentially change any field of the packet headers, in any layer. Some of them also modify packet payloads.

../../../_images/mbox3.png

The main difficulty in such a network environement is that the TCP state on the client and on the server are updated based on information carried out inside packets. When the information placed in these packets changes after their transmission by one of the communicating hosts, this can create strange problems. Several of the functions of the Multipath TCP were designed to cope with middlebox interference. Here are a few examples :

  • During the three-way handshake, the client sends the MP_CAPABLE option in the third ack to cope with a middlebox that could remove it from the SYN+ACK
  • The ADD_ADDR, REMOVE_ADDR and MP_JOIN option contain an address identifier to cope with Network Address Translation
  • The DSS option uses relative sequence numbers to cope with middleboxes that randomize the initial TCP sequence number
  • The DSS option maps of block of data from the bytestream onto the TCP subflow. The length field of the DSS option allows to cope with middleboxes (or fast NICs) that segment/reassemble packets
  • The DSS option contains a Checksum to cope with middleboxes that add/remove bytes in the payload

Multipath TCP and its implementation in the Linux kernel can cope with these interferences and others. This makes Multipath TCP very robust compared to older TCP extensions. An example with a strange middlebox was published in another blog post.

A detailed analysis of the reactions of Multipath TCP against those interferences was published in [HDP+13]. In some cases, Multipath TCP reacts by closing the subflow that passes through this middlebox. In other cases, it fallsback to regular TCP. A summary of this analysis may be found in the table below.

../../../_images/mbox4.png

If you suspect that there is a middlebox that interferes with Multipath TCP connections on a path, you can use tracebox [Gre] to detect the location of this middlebox. Examples of the utilisation of tracebox on Linux/MacOS and Android appeared on earlier blog posts.

References

[Gre]Detal Gregory. \texttt tracebox. http://www.tracebox.org.
[HDP+13]Benjamin Hesmans, Fabien Duchene, Christoph Paasch, Gregory Detal, and Olivier Bonaventure. Are TCP Extensions Middlebox-proof? In Proceedings of the 2013 Workshop on Hot Topics in Middleboxes and Network Function Virtualization (HotMiddlebox). 2013. URL: https://inl.info.ucl.ac.be/publications/are-tcp-extensions-middlebox-proof.html.
[HNR+11]M. Honda, Y. Nishida, C. Raiciu, A. Greenhalgh, M. Handley, and H. Tokuda. Is it still possible to extend TCP? In Proceedings of the 2011 ACM SIGCOMM conference on Internet Measurement Conference (IMC). 2011.