Advertising addresses with Multipath TCP

An important feature of Multipath TCP is that the communicating hosts can easily learn the addresses that can be used to reach their peers. Multipath TCP uses special TCP options to advertise this information. In this post, we look at the evolution of these options during the design of RFC 6824

The first MPTCP draft, draft-ford-mptcp-multiaddressed-00 defined four options to deal with addresses.

../../../_images/mptcp-0.png

The first option was called Add Address. It was intended to advertise one IP address owned by the host that sends it over a given Multipath TCP connection. This option contained an IP version field, to support IPv4 and IPv6, and an index. This index was intended to cope with NAT that could translate addresses in the IP header without translating information in the TCP options.

../../../_images/mptcp-addr-0.png

A companion option was the remove address option that simply contained the index of the address to be removed.

../../../_images/mptcp-rm-addr-0.png

In addition to these two options that have been slightly modified later in RFC 6824, draft-ford-mptcp-multiaddressed-00 also defined two options for implicit path management. The first one is Request-SYN. It was intended to request the peer to initiate a subflow towards the announced address. The Add Address option was intended to be purely informational and the Request-SYN was intended to trigger the establishment of a new subflow.

../../../_images/mptcp-req-syn.png

There was also a Request-FIN option that was intended to request the termination of a subflow.

images/mptcp-req-fin.png

These two options were removed in draft-ford-mptcp-multiaddressed-01 and only the Add and Remove address options were kept.

The working group draft, draft-ietf-mptcp-multiaddressed-00 introduced another two changes to these options. First, it became possible to advertise a specific port number together with an IP address. Second, the document included a discussion on adding a backup flag to indicate that the advertised address should be treated as a backup one. This bit became fully defined in draft-ietf-mptcp-multiaddressed-02. As of December 2018, it is unclear whether the ability to advertise a port number is used by a real implementation. Some measurements about the utilization of this option on www.multipath-tcp.org are discussed in [HTSB15] and [TCH+16].

../../../_images/mptcp-add-addr-1.png

Then, the backup bit was removed in draft-ietf-mptcp-multiaddressed-03. In draft-ietf-mptcp-multiaddressed-04, a new option called MP_PRIO was introduced to allow a host to dynamically change the priority (backup or not) of an address. This separation between the address advertisement and the announcement of the backup status was intended to provide more reactivity.

../../../_images/mp-prio.png

This option is sent by a receiver to indicate to its peer that it wants to change the backup status of the subflow over which this option is sent.

It should be noted that the MP_JOIN option that is used to create subflows contains a backup flag that allows to indicate whether a new subflow should be treated as a backup one or not. In contrast with the MP_PRIO option that is sent unreliably in a TCP option, the MP_JOIN option is always exchanged reliably, which guarantees that the communicating hosts know the backup status of newly established flows.

The discussion on advertising addresses continued within the IETF after the publication of RFC 6824. The Linux implementation supports this option and uses it to announce a new address as soon as this address is known. However, the reception of an ADD_ADDR option does not necessarily trigger the establishment of a subflow. This is under the responsibility of the path manager. The fullmesh patch manager running on a client tries to use all possible addresses, but a Linux server does not create subflows. Apple’s implementation does not support the ADD_ADDR option. They assume that the servers have a single IP address and that the client will anyway create subflows by using the MP_JOIN option.

One of the concerns with the ADD_ADDR option included in RFC 6824 was the risk of attacks where an attacker could inject his address in an existing Multipath TCP connection by sending a spoofed packet containing this option and valid TCP sequence numbers. This attack was described in the threats analysis RFC 7430. The new ADD_ADDR option was proposed at IETF88 and included in draft-ietf-mptcp-rfc6824bis-00. It includes a truncated HMAC that authenticates it with the keys exchanged during the initial handshake. It also ensures that this option is not modified by an on-path attacker that has not observed the initial handshake.

../../../_images/mptcp-add-addr-2.png

The discussion continued and draft-ietf-mptcp-rfc6824bis-01 introduced an Echo bit to provide a reliable delivery of this option.

../../../_images/mptcp-add-addr-3.png

As of December 2018, this option is included in the last version of draft-ietf-mptcp-rfc6824bis-12 but has not yet been deployed in the field.

References

[HTSB15]Benjamin Hesmans, Viet-Hoang Tran, Ramin Sadre, and Olivier Bonaventure. A first look at real multipath tcp traffic. In Traffic Monitoring and Analysis. 2015. URL: https://inl.info.ucl.ac.be/publications/first-look-real-multipath-tcp-traffic.html.
[TCH+16]Viet-Hoang Tran, Quentin De Coninck, Benjamin Hesmans, Ramin Sadre, and Olivier Bonaventure. Observing real multipath tcp traffic. Computer Communications, 2016. URL: https://inl.info.ucl.ac.be/publications/observing-real-multipath-tcp-traffic.html, doi:10.1016/j.comcom.2016.01.014.

Multipath TCP APIs

Shortly after the publication of RFC 6824, the IETF published RFC 6827. This RFC provides a first discussion of how a socket interface could be designed to support Multipath TCP. When RFC 6827 was published, none of the existing implementations included a socket API for Multipath TCP. The Linux implementation used the standard sockets with a few options and sysctls. This enabled Multipath TCP to be useable by any application, but without special means to control the utilisation of the subflows. This ability to support standard application remains one of the main advantages of this implementation. Once Multipath TCP has been added in the Linux kernel, all applications immediately benefit from its multipath capabilities.

However, there are some used cases where the applications know that they are using Multipath TCP and want to control some of its features such as the utilisation of the subflows of the packet scheduler. A first approach to provide this control was proposed in [HDB+15]. This solution exposes the Multipath TCP kernel API to user space applications through netlink messages.

../../../_images/netlink.png

Thanks to these messages, it is possible to design a userspace path manager librayr that control the utilisation of the subflows. This is important for use cases where one path is more costly than the other, e.g. on smartphones where the celullar interface is billed on a volume basis while the Wi-Fi interface is not metered. Several use cases and examples are described in [HDB+15].

A second possible approach is to extend the traditional socket API by using socket options that can be set or retrieved by the applications [HB16]. Six new socket options are proposed in this article and described later in draft-hesmans-mptcp-socket-03.

../../../_images/mptcp-socket.png

These socket options enable an application to control the creation and the removal of subflows during the lifetime of a Multipath TCP connection. Two use cases are described in this short paper : refreshing subflows on a regular basis and delaying the establishment of subflows. The patches that implement these socket options have been distributed on the mptcp-dev mailing list but not yet integrated in the official releases of the Multipath TCP implementation in the Linux kernel.

Apple has also defined a specific API to enable applications to interact with Multipath TCP. A simple example was provided in [HB16].

../../../_images/socket-apple.png

This API was private and not officially supported by Apple although there were examples in the Apple source code. When Apple announced Multipath TCP support for all iOS11 application, they also released a simple API that extends the URLSessionConfiguration property. This API does not provide a fine control to the utilization of the subflows as the socket API implemented on Linux, but it allows applications to select between the handover and interactive modes. Additional information is available from https://developer.apple.com/documentation/foundation/urlsessionconfiguration/improving_network_reliability_using_multipath_tcp

../../../_images/apple-url.png

References

[HB16](1, 2) Benjamin Hesmans and Olivier Bonaventure. An enhanced socket api for multipath tcp. In Proceeding ANRW ‘16 Proceedings of the 2016 Applied Networking Research Workshop. 2016. URL: https://inl.info.ucl.ac.be/publications/enhanced-socket-api-multipath-tcp.html.
[HDB+15](1, 2) Benjamin Hesmans, Gregory Detal, Sebastien Barre, Raphael Bauduin, and Olivier Bonaventure. Smapp : towards smart multipath tcp-enabled applications. In Proc. Conext 2015. Heidelberg, December 2015. URL: https://inl.info.ucl.ac.be/publications/smapp-towards-smart-multipath-tcp-enabled-applications.html.

Evolution of the documents produced by the MPTCP working group

Since its creation, the mptcp working group of the IETF has produced 7 documents that were published as RFCs and an eighth one is currently in last call. This post provides a brief description of these different documents as a starting point for someone who wants to start to look at multipath transport protocols.

../../../_images/rfcs.png

The working group has produced two Experimental RFCs and five Informational ones. The first two RFCs are RFC 6181 which discusses the security issues that were taken into account for the design of Multipath TCP and RFC 6182 which describes the basic architectural principles for Multipath TCP. RFC 6182 has already been discussed in a previous blog post. RFC 6181 was written by Marcelo Bagnulo and builds upon earlier work on a security analysis of Mobile IPv6 in RFC 4225 and shim6 RFC 4218. The key security issues that are discussed in this document are :

  • a flooding attack where an attacker who forces a server to send packets to a victim address by trying to force the server to use a subflow towards the victim. This attack was a concern for network layer protocols, but for Multipath TCP, this is not an issue since Multipath TCP validates the creation of the subflows with a three-way handshake and the MP_JOIN option.
  • a hijacking attack where an attacker leverages the address agility mechanisms of Multipath TCP to hijack an existing connection. One concern was the risk of an attacker being able to create a man-in-the-middle attack against existing Multipath TCP. This attack is prevented by the mechanism used by Multipath TCP to create subflows.
  • a discussion of time-shifted hijacking attacks

The threats analysis continued ad was later expanded in RFC 7430. This RFC introduces other types of attacks : an attack on ADD_ADDR which could allow an off-path attacker to create man-in-the-middle attack but under very unlikely circumstances, a denial of service attack on MP_JOIN, a SYN flood amplification attack and an eavesdropper that observes the initial handshake. These attacks were considered in the design of Multipath TCP or their implementations.

The other Informational RFCs are RFC 6897 which discusses API considerations and RFC 8041 which describes the known use cases where Multipath TCP has been deployed. Some of these use cases are also discussed in [BS16].

The three main documents produced by the mptcp working group are RFC 6356 which defines the coupled congestion control scheme and RFC 6824. As their publication date suggests, the congestion control scheme was stable much earlier and the protocol specification.

Experimental and Standards-track RFCs contain MUST, SHOULD and other keywords defined in RFC 2119. As RFC 6356 specifies a congestion control scheme, it only include two MUST keywords. However, RFC 6824 provides a precise protocol specification that leverages these keywords.

../../../_images/draft-ietf-mptcp-multiaddressed_keywords.png

The above figure, plotted with a script developed by Maxime Piraux for [PDCB18], shows the evolution of the utilization of the RFC 2119 keywords in the different versions of draft-ietf-mptcp-multiaddressed. There were three main phases in the utilization of these keywords: a low usage in the first four drafts, then a sudden increase in draft-ietf-mptcp-multiaddressed-03 that included changes in the MP_CAPABLE option, the addition of an address identifier in MP_PRIO and answers to review comments a huge list of changes.

It is interesting to observe the evolution of draft-ietf-mptcp-rfc6824bis

../../../_images/draft-ietf-mptcp-rfc6824bis_keywords.png

The above figure shows that when looking at the RFC 2119 keywords, the specification did not change a lot compared to RFC 6824. Most of the changes were clarifications except for the redefinition of the MP_CAPABLE option as discussed in a previous blog post.

In contrast, the transport part of QUIC, defined in draft-ietf-quic-transport appears to be much more complex when counting the RFC 2119 keywords and the specification is not yet finished.

../../../_images/draft-ietf-quic-transport_keywords.png

References

[BS16]Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/.
[PDCB18]Maxime Piraux, Quentin De Coninck, and Olivier Bonaventure. Observing the evolution of quic implementations. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC, EPIQ‘18, 8–14. New York, NY, USA, 2018. ACM. URL: https://quic-tracker.info.ucl.ac.be/blog/results/paper/2018/11/19/epiq-18-paper-accepted.html, doi:10.1145/3284850.3284852.

Apple uses Multipath TCP

The initial specification for Multipath TCP was published in January 2013 RFC 6824. Apple had participated to some of the discussions during the IETF meetings before, but never announced a deployment. Shortly after the publication of RFC 6824, Phil Eardley published a blank internet draft, draft-eardley-mptcp-implementations-survey-01 that explicitly asked questions to implementers.

../../../_images/survey.png

Four implementations were disclosed during the summer of 2013:

  • the stable Linux implementation discussed on page 11
  • an ongoing implementation on FreeBSD discussed on page 19
  • an anonymous implementation discussed on page 23
  • on implementation on Citrix load balancers discussed on page 31

Five years after, it is interesting to look at the characteristics of this anonymous implementation.

  • This implementation only supports client-initiated subflows
  • It uses 4 bytes DSN as a default, but can support 8 bytes DSN
  • The support for ADD_ADDR and REMOVE_ADDR was described as : It does not support sending ADD_ADDR or processing ADD_ADDR as it is considered a security risk. Also, we only have a client side implementation at the moment which always initiates the sub flows. The remote end does not send ADD_ADDR in our configuration. The client can send REMOVE_ADDR however when one of the established sub flow’s source address goes away. The client ignores incoming REMOVE_ADDR options also.
  • It does not implement the coupled congestion control defined in RFC 6356
  • It uses a private API and not the socket API proposed in RFC 6897
  • The proposed deployment is described as follows : MPTCP in mobile environments is very powerful when used in the active/backup mode. Since the network interfaces available on mobile devices have different cost characteristics as well as different bring up and power usage characteristics, it is not useful to share load across all available network interfaces - at least not currently. Providing session continuity across changing network environments is the key deployment scenario.

In September 2013, Apple launched iOS7 that included support for Multipath TCP. Apple’s motivation for using Multipath TCP on iOS have been explained in details in [BS16]:

Siri is the digital assistant in Apple’s iOS and macOS operating systems. Because speech recognition requires tremendous processing power, Siri streams spoken commands to Apple’s datacenter for speech recognition; the result is sent back to the smartphone. Although the duration of a user’s interaction with Siri is relatively short, Siri’s usage pattern made this data transfer a perfect client for MPTCP.

Many people use Siri while walking or driving. As they move farther away from a WiFi access point, the TCP connection used by Siri to stream its voice eventually fails, resulting in error messages.

To address this issue, Apple has been using MPTCP—and benefiting from its handover capabilities—since its iOS 7 release. When a user issues a Siri voice command, iOS establishes an MPTCP connection over WiFi and cellular. If the phone loses connectivity to the WiFi access point, traffic is handed over to the cellular interface. A WiFi connection that is still in sight of an access point can have a channel become so lossy that barely any segments can be transmitted. In this case, another retransmission timeout happens and iOS retransmits the traffic over the cellular link.

The article continues with additional information that describes how Apple has tuned Multipath TCP to this specific use case. A description of the Multipath TCP handshake used by Siri has been published in a previous blog post.

While Multipath TCP was part of iOS, it was only used by Apple’s own Siri applications. The regular applications could not leverage the benefits of Multipath TCP. This changed in 2017 with the launch of iOS11. During WDC2017, Christoph Paasch and his colleagues announced that any application would be able to use Multipath TCP on iOS11.

../../../_images/christoph.png

A detailed summary of these announcements appear on the tessares blog. iOS11 supports two modes of operation : Handover and Interactive.

Connection starts over the WiFi link and no packet is sent over the cellular interface. If the signal gets worse, a new TCP subflow will be created on the cellular interface automatically. The cellular subflow will be removed once the user is back in a WiFi network.

../../../_images/handover-apple.png

The interactive mode establishes both WiFi and cellular subflows for each Multipath TCP connection, even if the WiFi network appears to be working well. The objective of this mode is to reduce latency. The Multipath TCP scheduler will select the flow that provides the lowest latency.

../../../_images/interactive.png

Since the publication of iOS11, some applications have started to use Multipath TCP. One of them is the Multipath Tester, an application written by Quentin De Coninck that allows to compare the performance of Multipath TCP and Multipath QUIC on iOS11 [CB18]. You can download it from https://itunes.apple.com/us/app/multipathtester/id1351286809

../../../_images/mptester.png

References

[BS16]Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/.
[CB18]Quentin De Coninck and Olivier Bonaventure. Observing network handovers with multipath TCP. In Proceedings of the ACM SIGCOMM 2018 Conference on Posters and Demos, SIGCOMM 2018, Budapest, Hungary, August 20-25, 2018, 54–56. 2018. URL: https://multipath-quic.org/multipathtester/2018/08/28/sigcomm-poster.html, doi:10.1145/3234200.3234214.