Which servers use Multipath TCP ?
Since the publication of RFC 6824, Multipath TCP has been deployed for several use cases described in RFC 8041 and [BS16]. However, there has been no public study of the deployment of Multipath TCP on servers.
A first study appeared in [MHFB15]. They used zmap to send SYN packets with the MP_CAPABLE option and checked whether they replied with the MP_CAPABLE option. They used 452,008 unique IP addresses corresponding to a bit less than 2 million domains. Their results are summarised in the table below (source [MHFB15]).
Unfortunately, there was a flaw in the methodology. Checking the presence of the MP_CAPABLE option in the SYN+ACKis not sufficient as some middleboxes simply echo in the SYN+ACK any option contained in the SYN even if they do not understand it. The dataset was later corrected and updated. Any zmap scan must check that a returned TCP option in a SYN+ACK is not simply echoed by a middlebox. For Multipath TCP, this means that the returned MP_CAPABLE option must differ from the option sent in the SYN.
Since the publication of this paper, no other study using zmap has been published. Several research groups use zmap to carry out measurement studies. One of them is the netray Internet Observatory at RWTH Aachen University. They have used zmap to study the deployment of HTTP/2, QUIC or the initial TCP congestion window. I recently discussed with Jan Ruth and Oliver Hohlfeld who had some interesting data. In August 2015, a scan of the entire IPv4 addressing space revealed 921 Multipath TCP servers, with about one sixth of them hosted by Apple. At the same time, about 24.000 addresses returned in the SYN+ACK an MP_CAPABLE option that was exactly the same as the one sent in the SYN packet.
They proposed to launch another zmap scan last week over the entire IPv4 addressing space on port 443. 88484 addresses replied with an MP_CAPABLE option. Out of these, 77384 simply returned exactly the same MP_CAPABLE option as the one included in the SYN packet. There are thus more middleboxes that echo unsupported TCP options than in August 2015. The interesting point of their study is that 11k addresses replied with a valid MP_CAPABLE option. This indicates an important growth in the deployment of Multipath TCP on servers. This analysis ignores the utilisation of Multipath TCP on clients such as iPhones and other smartphones, but also in Hybrid Access Networks since none of those devices respond to SYN packets.
The exact usage of these servers is not known. If you manage one of them, we’d be interested in knowing more about your use case to document it in revisions of RFC 8041. If it receives a lot of traffic, we’d also be interested in analysing packet traces in more details to extend the measurements published in [CBHB16a][CBHB16b][HTSB15][TCH+16].
References
| [BS16] | Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/. |
| [CBHB16a] | Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. A first analysis of multipath tcp on smartphones. In 17th International Passive and Active Measurements Conference, volume 17. Springer, March-April 2016. URL: https://inl.info.ucl.ac.be/publications/first-analysis-multipath-tcp-smartphones.html. |
| [CBHB16b] | Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. Observing real smartphone applications over multipath tcp. IEEE Communications Magazine, Network Testing Series, March 2016. URL: https://inl.info.ucl.ac.be/publications/observing-real-smartphone-applications-over-multipath-tcp.html. |
| [HTSB15] | Benjamin Hesmans, Viet-Hoang Tran, Ramin Sadre, and Olivier Bonaventure. A first look at real multipath tcp traffic. In Traffic Monitoring and Analysis. 2015. URL: https://inl.info.ucl.ac.be/publications/first-look-real-multipath-tcp-traffic.html. |
| [MHFB15] | (1, 2) Olivier Mehani, Ralph Holz, Simone Ferlin, and Roksana Boreli. An early look at multipath tcp deployment in the wild. In Proceedings of the 6th International Workshop on Hot Topics in Planet-Scale Measurement, HotPlanet ‘15, 7–12. New York, NY, USA, 2015. ACM. URL: http://doi.acm.org/10.1145/2798087.2798088, doi:10.1145/2798087.2798088. |
| [TCH+16] | Viet-Hoang Tran, Quentin De Coninck, Benjamin Hesmans, Ramin Sadre, and Olivier Bonaventure. Observing real multipath tcp traffic. Computer Communications, 2016. URL: https://inl.info.ucl.ac.be/publications/observing-real-multipath-tcp-traffic.html, doi:10.1016/j.comcom.2016.01.014. |
Advertising addresses with Multipath TCP
An important feature of Multipath TCP is that the communicating hosts can easily learn the addresses that can be used to reach their peers. Multipath TCP uses special TCP options to advertise this information. In this post, we look at the evolution of these options during the design of RFC 6824
The first MPTCP draft, draft-ford-mptcp-multiaddressed-00 defined four options to deal with addresses.
The first option was called Add Address. It was intended to advertise one IP address owned by the host that sends it over a given Multipath TCP connection. This option contained an IP version field, to support IPv4 and IPv6, and an index. This index was intended to cope with NAT that could translate addresses in the IP header without translating information in the TCP options.
A companion option was the remove address option that simply contained the index of the address to be removed.
In addition to these two options that have been slightly modified later in RFC 6824, draft-ford-mptcp-multiaddressed-00 also defined two options for implicit path management. The first one is Request-SYN. It was intended to request the peer to initiate a subflow towards the announced address. The Add Address option was intended to be purely informational and the Request-SYN was intended to trigger the establishment of a new subflow.
There was also a Request-FIN option that was intended to request the termination of a subflow.
These two options were removed in draft-ford-mptcp-multiaddressed-01 and only the Add and Remove address options were kept.
The working group draft, draft-ietf-mptcp-multiaddressed-00 introduced another two changes to these options. First, it became possible to advertise a specific port number together with an IP address. Second, the document included a discussion on adding a backup flag to indicate that the advertised address should be treated as a backup one. This bit became fully defined in draft-ietf-mptcp-multiaddressed-02. As of December 2018, it is unclear whether the ability to advertise a port number is used by a real implementation. Some measurements about the utilization of this option on www.multipath-tcp.org are discussed in [HTSB15] and [TCH+16].
Then, the backup bit was removed in draft-ietf-mptcp-multiaddressed-03. In draft-ietf-mptcp-multiaddressed-04, a new option called MP_PRIO was introduced to allow a host to dynamically change the priority (backup or not) of an address. This separation between the address advertisement and the announcement of the backup status was intended to provide more reactivity.
This option is sent by a receiver to indicate to its peer that it wants to change the backup status of the subflow over which this option is sent.
It should be noted that the MP_JOIN option that is used to create subflows contains a backup flag that allows to indicate whether a new subflow should be treated as a backup one or not. In contrast with the MP_PRIO option that is sent unreliably in a TCP option, the MP_JOIN option is always exchanged reliably, which guarantees that the communicating hosts know the backup status of newly established flows.
The discussion on advertising addresses continued within the IETF after the publication of RFC 6824. The Linux implementation supports this option and uses it to announce a new address as soon as this address is known. However, the reception of an ADD_ADDR option does not necessarily trigger the establishment of a subflow. This is under the responsibility of the path manager. The fullmesh patch manager running on a client tries to use all possible addresses, but a Linux server does not create subflows. Apple’s implementation does not support the ADD_ADDR option. They assume that the servers have a single IP address and that the client will anyway create subflows by using the MP_JOIN option.
One of the concerns with the ADD_ADDR option included in RFC 6824 was the risk of attacks where an attacker could inject his address in an existing Multipath TCP connection by sending a spoofed packet containing this option and valid TCP sequence numbers. This attack was described in the threats analysis RFC 7430. The new ADD_ADDR option was proposed at IETF88 and included in draft-ietf-mptcp-rfc6824bis-00. It includes a truncated HMAC that authenticates it with the keys exchanged during the initial handshake. It also ensures that this option is not modified by an on-path attacker that has not observed the initial handshake.
The discussion continued and draft-ietf-mptcp-rfc6824bis-01 introduced an Echo bit to provide a reliable delivery of this option.
As of December 2018, this option is included in the last version of draft-ietf-mptcp-rfc6824bis-12 but has not yet been deployed in the field.
References
| [HTSB15] | Benjamin Hesmans, Viet-Hoang Tran, Ramin Sadre, and Olivier Bonaventure. A first look at real multipath tcp traffic. In Traffic Monitoring and Analysis. 2015. URL: https://inl.info.ucl.ac.be/publications/first-look-real-multipath-tcp-traffic.html. |
| [TCH+16] | Viet-Hoang Tran, Quentin De Coninck, Benjamin Hesmans, Ramin Sadre, and Olivier Bonaventure. Observing real multipath tcp traffic. Computer Communications, 2016. URL: https://inl.info.ucl.ac.be/publications/observing-real-multipath-tcp-traffic.html, doi:10.1016/j.comcom.2016.01.014. |
Multipath TCP APIs
Shortly after the publication of RFC 6824, the IETF published RFC 6827. This RFC provides a first discussion of how a socket interface could be designed to support Multipath TCP. When RFC 6827 was published, none of the existing implementations included a socket API for Multipath TCP. The Linux implementation used the standard sockets with a few options and sysctls. This enabled Multipath TCP to be useable by any application, but without special means to control the utilisation of the subflows. This ability to support standard application remains one of the main advantages of this implementation. Once Multipath TCP has been added in the Linux kernel, all applications immediately benefit from its multipath capabilities.
However, there are some used cases where the applications know that they are using Multipath TCP and want to control some of its features such as the utilisation of the subflows of the packet scheduler. A first approach to provide this control was proposed in [HDB+15]. This solution exposes the Multipath TCP kernel API to user space applications through netlink messages.
Thanks to these messages, it is possible to design a userspace path manager librayr that control the utilisation of the subflows. This is important for use cases where one path is more costly than the other, e.g. on smartphones where the celullar interface is billed on a volume basis while the Wi-Fi interface is not metered. Several use cases and examples are described in [HDB+15].
A second possible approach is to extend the traditional socket API by using socket options that can be set or retrieved by the applications [HB16]. Six new socket options are proposed in this article and described later in draft-hesmans-mptcp-socket-03.
These socket options enable an application to control the creation and the removal of subflows during the lifetime of a Multipath TCP connection. Two use cases are described in this short paper : refreshing subflows on a regular basis and delaying the establishment of subflows. The patches that implement these socket options have been distributed on the mptcp-dev mailing list but not yet integrated in the official releases of the Multipath TCP implementation in the Linux kernel.
Apple has also defined a specific API to enable applications to interact with Multipath TCP. A simple example was provided in [HB16].
This API was private and not officially supported by Apple although there were examples in the Apple source code. When Apple announced Multipath TCP support for all iOS11 application, they also released a simple API that extends the URLSessionConfiguration property. This API does not provide a fine control to the utilization of the subflows as the socket API implemented on Linux, but it allows applications to select between the handover and interactive modes. Additional information is available from https://developer.apple.com/documentation/foundation/urlsessionconfiguration/improving_network_reliability_using_multipath_tcp
References
| [HB16] | (1, 2) Benjamin Hesmans and Olivier Bonaventure. An enhanced socket api for multipath tcp. In Proceeding ANRW ‘16 Proceedings of the 2016 Applied Networking Research Workshop. 2016. URL: https://inl.info.ucl.ac.be/publications/enhanced-socket-api-multipath-tcp.html. |
| [HDB+15] | (1, 2) Benjamin Hesmans, Gregory Detal, Sebastien Barre, Raphael Bauduin, and Olivier Bonaventure. Smapp : towards smart multipath tcp-enabled applications. In Proc. Conext 2015. Heidelberg, December 2015. URL: https://inl.info.ucl.ac.be/publications/smapp-towards-smart-multipath-tcp-enabled-applications.html. |
Evolution of the documents produced by the MPTCP working group
Since its creation, the mptcp working group of the IETF has produced 7 documents that were published as RFCs and an eighth one is currently in last call. This post provides a brief description of these different documents as a starting point for someone who wants to start to look at multipath transport protocols.
The working group has produced two Experimental RFCs and five Informational ones. The first two RFCs are RFC 6181 which discusses the security issues that were taken into account for the design of Multipath TCP and RFC 6182 which describes the basic architectural principles for Multipath TCP. RFC 6182 has already been discussed in a previous blog post. RFC 6181 was written by Marcelo Bagnulo and builds upon earlier work on a security analysis of Mobile IPv6 in RFC 4225 and shim6 RFC 4218. The key security issues that are discussed in this document are :
- a flooding attack where an attacker who forces a server to send packets to a victim address by trying to force the server to use a subflow towards the victim. This attack was a concern for network layer protocols, but for Multipath TCP, this is not an issue since Multipath TCP validates the creation of the subflows with a three-way handshake and the MP_JOIN option.
- a hijacking attack where an attacker leverages the address agility mechanisms of Multipath TCP to hijack an existing connection. One concern was the risk of an attacker being able to create a man-in-the-middle attack against existing Multipath TCP. This attack is prevented by the mechanism used by Multipath TCP to create subflows.
- a discussion of time-shifted hijacking attacks
The threats analysis continued ad was later expanded in RFC 7430. This RFC introduces other types of attacks : an attack on ADD_ADDR which could allow an off-path attacker to create man-in-the-middle attack but under very unlikely circumstances, a denial of service attack on MP_JOIN, a SYN flood amplification attack and an eavesdropper that observes the initial handshake. These attacks were considered in the design of Multipath TCP or their implementations.
The other Informational RFCs are RFC 6897 which discusses API considerations and RFC 8041 which describes the known use cases where Multipath TCP has been deployed. Some of these use cases are also discussed in [BS16].
The three main documents produced by the mptcp working group are RFC 6356 which defines the coupled congestion control scheme and RFC 6824. As their publication date suggests, the congestion control scheme was stable much earlier and the protocol specification.
Experimental and Standards-track RFCs contain MUST, SHOULD and other keywords defined in RFC 2119. As RFC 6356 specifies a congestion control scheme, it only include two MUST keywords. However, RFC 6824 provides a precise protocol specification that leverages these keywords.
The above figure, plotted with a script developed by Maxime Piraux for [PDCB18], shows the evolution of the utilization of the RFC 2119 keywords in the different versions of draft-ietf-mptcp-multiaddressed. There were three main phases in the utilization of these keywords: a low usage in the first four drafts, then a sudden increase in draft-ietf-mptcp-multiaddressed-03 that included changes in the MP_CAPABLE option, the addition of an address identifier in MP_PRIO and answers to review comments a huge list of changes.
It is interesting to observe the evolution of draft-ietf-mptcp-rfc6824bis
The above figure shows that when looking at the RFC 2119 keywords, the specification did not change a lot compared to RFC 6824. Most of the changes were clarifications except for the redefinition of the MP_CAPABLE option as discussed in a previous blog post.
In contrast, the transport part of QUIC, defined in draft-ietf-quic-transport appears to be much more complex when counting the RFC 2119 keywords and the specification is not yet finished.
References
| [BS16] | Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/. |
| [PDCB18] | Maxime Piraux, Quentin De Coninck, and Olivier Bonaventure. Observing the evolution of quic implementations. In Proceedings of the Workshop on the Evolution, Performance, and Interoperability of QUIC, EPIQ’18, 8–14. New York, NY, USA, 2018. ACM. URL: https://quic-tracker.info.ucl.ac.be/blog/results/paper/2018/11/19/epiq-18-paper-accepted.html, doi:10.1145/3284850.3284852. |