Multipath TCP on iOS11 : A closer look at the TCP Options

Multipath TCP uses a variety of TCP options to use different paths simultaneously. Several Multipath TCP options are defined in RFC6824 :

  • subtype 0x0: MP_CAPABLE
  • subtype 0x1: MP_JOIN
  • subtype 0x2: DSS
  • subtype 0x3: ADD_ADDR
  • subtype 0x4: REMOVE_ADDR
  • subtype 0x5: MP_PRIO
  • subtype 0x6: MP_FAIL
  • subtype 0x7: MP_FASTCLOSE

In this blog post, we explore in more details the packet trace collected on an iPhone using iOS11 beta. We start our analysis with the three-way handshake. The trace contains one Multipath TCP connection. Recent versions of Wireshark support Multipath TCP and we use the tcp.options.mptcp.subtype==0 filter to match all the packets that contain the MP_CAPABLE option. This option only appears in the three packets of the initial three-way handshake. Let us first analyse the SYN sent by the iPhone. In our test over an LTE network, iOS11 beta2 advertises the following options:

  • MSS set to 1410 bytes. This is a relatively small value that was probably chosen to reduce the risk of fragmentation or Path MTU discovery problems since cellular networks often use tunnels internally
  • Selective Acknowledgements are proposed
  • The Window scale factor is set to 6 and the iPhone advertises a 64Kbytes window.
  • The Timestamp option is used as well.
  • The MP_CAPABLE option sent by the iPhone does not request the utilisation of the DSS checksum. The DSS checksum was introduced in RFC6824 to detect middlebox interference. Previous versions of iOS did not use this checksum to support Siri because Siri ran over HTTPS and this prevents most middlebox interference. However, when Multipath TCP is used to support a protocol such as HTTP, there is a risk of interference from middleboxes that inject HTTP headers. If you plan to use Multipath TCP on iOS11, you should probably rely on HTTPS and forget HTTP for other reasons than Multipath TCP.
../../../_images/syn.png

The server, in this trace the Linux implementation running on multipath-tcp.org replies with Selective Acknowledgements, Timestamps, a Window Scaling factor set to 7 and requires the utilisation of the DSS Checksum.

../../../_images/synack.png

The MP_CAPABLE option contained in the third ACK sent by the iPhone confirms that the iPhone will use the DSS checksum for this connection as requested by the server.

../../../_images/thirdack.png

The utilisation of the DSS Checksum is clearly visible in the first data packet that is sent by the iPhone. It uses 32 bits long Data sequence numbers and data acknowledgement numbers.

../../../_images/dss1.png

The first data packet returned by the Linux server is shown below. It also uses 32 bits data sequence and data acknowledgement numbers.

../../../_images/dss2.png

With iOS11 beta2, the iPhone uses the MP_PRIO option and sets the cellular subflow as a backup subflow. This is immediately visible in the fourth packet of the trace that is shown below.

../../../_images/mpprio.png

Apple has already explained earlier that they do not use the ADD_ADDR option because their stack is focussed on clients and they do not see a benefit in advertising client addresses since those are often behind a NAT or firewall. We did not observe ADD_ADDR or REMOVE_ADDR in our first trace.

The MP_JOIN option is used to create subflows. In our trace, this happens at time 4.74 when we enable the WiFi interface. The MP_JOIN option contains the token advertise by the server in the MP_CAPABLE option and its backup flag is reset. This indicates that the WiFi subflow is preferred to the cellular flow that was initially created. It is interesting to note that iOS11 beta advertises a longer MSS over the WiFi interface than over the cellular one. The same window scaling factor (6) is used.

../../../_images/mpjoin.png

We did not observe MP_FASTCLOSE in this trace.

We’ll discuss MP_FAIL in another post since it is related to fallbacks to TCP.

MPTCP experiments on iOS 11 beta

MPTCP support has been announced for iOS 11 during wwwdc2017. The developer documentation presents a new instance property called multipathServiceType inside the URLSessionConfiguration class that can be set to one of the constants specified in MultipathServiceType enumeration, which is also in the URLSessionConfiguration class. The enumeration contains four constants and the documentation has a small description for each constant :

  1. none : The default service type indicating that Multipath TCP should not be used.
  2. handover : A Multipath TCP service that provides seamless handover between Wi-Fi and cellular in order to preserve the connection.
  3. interactive : A service whereby Multipath TCP attempts to use the lowest-latency interface.
  4. aggregate : A service that aggregates the capacities of other Multipath options in an attempt to increase throughput and minimize latency.

The code bellow shows a simple example of usage:

let config = URLSessionConfiguration.ephemeral

config.multipathServiceType = URLSessionConfiguration.MultipathServiceType.handover
let session = URLSession(configuration: config)

let url = URL(string: "http://multipath-tcp.org/data/uml/vmlinux_64")

let task = session.dataTask(with: url!, completionHandler:{...})

task.resume()

We will present experiments done with iOS in a series of posts on this blog. In our first experiment, we use the handover service type. We start the connection with the wifi interface down and after a few seconds, we turn on the wifi interface. The trace of the connection is available here. We use mptcptrace to see how the subflows are used. Let’s take a look at the Multipath-TCP sequence numbers over time :

../../../_images/sequence5.png

As expected, the connection starts on the mobile interface because it is the only interface available at that time. When the wifi interface becomes available, around five seconds after the start of the connection, all the traffic is immediately sent to the wifi subflow.

Let’s take a closer look at what happens during the transition around five seconds after the start of the connection:

../../../_images/zoom1.png

On this graph, MPTCP acknowledgements are pictured as blue crosses. We can see on this zoom, on the left upper corner, that the client receives out-of-sequence (from Multipath-TCP’s perspective) packets during the transition. This is due to the fact that iOS tries to terminate the connection as soon as possible on the mobile interface and the server does not know yet that it should not be used anymore. Starting from packets 4647 in the trace, we can see the zero window advertisement and resets sent by the iPhone on the mobile subflow. Once the server detects that some packets will not arrive on the mobile subflow, when it receives the reset, it reinjects the packets on the wifi subflow. During the time of the reinjections, out-of-order packets are kept in the out-of-order queue of MPTCP on the client side. To observe this out-of-sequence queue, we zoom on the right top corner of the graph :

../../../_images/zoom2.png

On this graph, we can observe the MPTCP ACKs that cover the out-of-sequence packets received earlier. In particular we can observe a hole in the middle of the graph. If we zoom on other parts of the graph we can see several holes like this one.

This concludes our first analysis of Multipath-TCP on iOS. Stay tuned for more detailed analysis and tests. In next posts, we will discuss other Multipath TCP services offered by iOS11.

The “Experimental” status of Multipath TCP

Multipath TCP is defined in RFC 6824 and I recently heard feedback from someone working for industry who mentioned that Multipath TCP should not be considered for deployment given its Experimental status. I was surprised by this comment and I think that it would be useful to clarify some facts about the maturity of Multipath TCP.

First, from a administrative viewpoint, the Experimental status of Multipath TCP was decided at the creation of the IETF MPTCP working group. At that time, it was unclear whether it would be even possible to specify a protocol like Multipath TCP and the IESG wanted to encourage experiments with the new protocol. By selecting this option , the IESG prepared a future standardisation of the protocol and this is happening right now with the definition of a standards-track version of Multipath TCP in RFC6824bis . According to the milestones of the IETF MPTCP working group, this revision should be ready in 2017.

Second, from a technical viewpoint, the maturity of a protocol cannot be inferred from the status of its specification. The best way to measure this maturity is to observe the interoperable implementations and the deployment of the protocol. From these two viewpoints, Multipath TCP is a clear success. There are endhost implementations on Linux, FreeBSD, Apple iOS, MacOS and Oracle Solaris. Multipath TCP is also supported on various middleboxes including Citrix Netscaler, F5 BIG-IP LTM and Ericsson.

From a deployment viewpoint, Multipath TCP is also a huge success. Hundreds of millions of users of Apple devices (iPhone, iPad, laptops) use Multipath TCP every time they use the Siri voice recognition application. In Korea, a dozen of models of high-end smartphones from Samsung and LG include a port of the reference implementation of Multipath TCP in the Linux kernel and use SOCKS proxies to bond WiFi and fast LTE. Several network operators provide those proxies as a commercial service. Other companies such as Swisscom or OVH also rely on SOCKS proxies to bond different types of links together. Another emerging use case are hybrid access networks. In various countries, network operators are require to provide fast broadband services, even in rural areas where deploying fiber is too expensive. Many of these operators want to combine their xDSL and LTE networks in order to improve the bandwidth to their customers. Tessares has already deployed a pilot hybrid access network solution that leverages Multipath TCP in Belgium.

Multipath TCP projects during the IETF97 Hackathon

The IETF organised a hackathon during the weekend before the IETF‘97 meeting in Seoul. There are already several large scale deployments of Multipath TCP. However, these deployments focus on very specific utilisations of Multipath TCP for special applications or through various forms of proxies.

Recently, Benjamin Hesmans has released an enhanced socket API for Multipath TCP . This API has the potential of enabling new use cases for Multipath TCP by allowing application developpers to control the establishment and the utilisation of the Multipath TCP subflows. To understand how this new API could be used, we organised a remote hackathon at Ecole Polytechnique de Louvain. We had two teams working on Multipath TCP during the IETF‘97 Hackathon. In Seoul, 5 IETFers, including three PhD students from the IP Networking Lab worked in Seoul and 25 students worked in Louvain-la-Neuve on this new socket API.

../../../_images/hackathon.png

These two teams received the best overall award from the organisers of the IETF‘97 Hackathon in Seoul for their effort.

The Seoul team, composed of Benjamin Hesmans, Fabien Duchene, Olivier Tilmans, SungHoon Seo and François Serman worked on developing a library that can be pre-loaded before launching an unmodified application to use the new socket API to control how this application uses the underlying Multipath TCP subflows. This is described in these slides

In Louvain-la-Neuve, eight teams worked on different use cases.

Two groups of students worked on porting the Multipath TCP socket API to other langages than C. They have created prototype code for Java and Ruby. The other teams worked on curl, lighttpd, Openssh, ipef3 and nc.

Grégory Vander Schueren, Raphaël Bauduin and Thibault Gérondal worked on modifying Ruby to support the new Multipath TCP socket API. They obtained running code to support some of the new socket operations directly from ruby. Their work is summarised in these slides

../../../_images/ruby.jpg

Guillaume Demaude and Pierre Ortegat have analysed the problem of supporting the new Multipath TCP socket API in Java. It turned out that the Socket class in Java had not been designed to be extended. They have thus written static methods that implement the new socket API. Their results are in summarised in these slides and their code is available from https://github.com/reirep/matcp-java.git

../../../_images/java.jpg

Hoang Tran Viet, Remi Chauvenne and Thibault Libioulle have worked on iperf3, a throughput measurement tool. They have added the support for Multipath TCP inside iperf3 and modified the application to exchange the addresses of the client and of the server that are used to perform the tests. Their results are summarised in these slides

../../../_images/iperf3.jpg

Charles-Henry Bertrand and Sylvain Dassier have explored how to modify the netcat testing tool to support Multipath TCP. Their results are summarised in these slides

../../../_images/nc.jpg

Maxime Beugom, Antoine Denauw, Alexandre Dubray, Julien Gomez and Julian Roussieau have worked on Openssh. Their prototype controls changes the underlying subflows after the transmission of a number of bytes or after some time. This prototype demonstrates that by influencing the underlying subflows a security application can select different paths and thus counter on-path attacks. Their results are summarised in these slides Their code is posted on https://github.com/Derwaan/openssh-portable

../../../_images/ssh.jpg

Arnaud Dethise and Jacob Eliat-Eliat have modified curl to only create Multipath TCP subflows on connections that carry a sufficient number of bytes or last a sufficient time. Experiments have shown that Multipath TCP does not bring benefits for very short flows and this demonstrates how an application can defer the establishment of subflows. Their results are summarised in these slides and their prototype code is available from https://github.com/adethise/curl/tree/mptcp

../../../_images/curl.jpg

Maxime Andries, Pablo Gonzalez Alvarez and Antoine Lambot have explored the possibility of creating subflows from the server. For this, they started from the lighttpd server and have modified it to create subflows when the web object returned by the server is large enough. Their results are summarised in these slides

../../../_images/lighttpd.jpg

Alexis Clarembeau has explored the possibility of developing a higher-level API for Multipath TCP that exposes a more abstract interface to the application. His results are summarised in these slides

../../../_images/alexis.jpg