Apple Music on iOS13 uses Multipath TCP through load-balancers

Apple Music on iOS13 uses Multipath TCP through load-balancers

Since the publication of RFC6824 in 2013, interest in Multipath TCP has continued to grow and various use cases have been deployed. Apple uses Multipath TCP for its Siri voice recognition application since 2013 to support seamless handovers while walking. Tessares uses Multipath TCP to deploy Hybrid Access Networks that combine xDSL and LTE to provide faster Internet access services in rural areas. Samsung, LG and Huawei use Multipath TCP on their Android smartphones to combine Wi-Fi and 4G. Recently, 3GPPP has selected Multipath TCP to support Wi-Fi/5G co-existence in future 5G networks and a first prototype has been demonstrated.

Despite these growing deployments, web hosting companies and CDNs have complained that Multipath TCP was difficult to deploy because they assume that load balancers would need to be changed to terminate Multipath TCP connections.

It turns out that it is possible to support Multipath TCP on servers with todays’ load-balancers. Fabien Duchêne proposed and evaluated this solution in the Making Multipath TCP Friendlier to Load-Balancers and Anycast paper that he presented at ICNP’17. A simpler load-balancer is illustrated in the figure below. The load-balancer uses IP address 1.2.3.4 and forwards connections to the servers behind it, selecting them e.g. based on a hash function.

../../../_images/lb.png

With Multipath TCP, this simple approach does not work anymore as the second subflow from the client could be hashed to a different server than the one of the initial subflow.

../../../_images/lb3.png

The solution proposed by Fabien Duchêne is both simple and efficient. The load-balancer has a public IP address that is advertised by the DNS. Furthermore, each server that resides behind this load balancer has it own public IP address. When a client contacts the load-balanced service, its SYN packet reaches the load-balancer that selects one of the servers in the pool. The server confirms the establishment of the connection using the load-balanced IP address. As soon as the Multipath TCP connection is established, the server sends a packet containing its own IP address inside an ADD_ADDR option. Thanks to this address, the client can establish the subsequent subflows directly towards the server address, effectively bypassing the load-balancer.

../../../_images/lb4.png

Given its benefits, this solution has been included in the standards-track version of Multipath TCP that is being finalised by the IETF. However, it had not yet been deployed at a large scale, until the release of iOS13 by Apple in September 2019. Given the benefits of Multipath TCP for Siri, Apple has decided to also use Multipath TCP for its Apple Maps and Apple Music applications. These two applications are often used while walking and thus suffered from interruptions during Wi-Fi/cellular handovers. A closer look at a packet trace collected from an iPhone using Apple Music shows interesting results.

First, Apple Music uses Multipath TCP as shown by the options contained in the SYN packet. It is interesting to point out that both the iPhone and the server use the first version of Multipath TCP defined in RFC6824.

../../../_images/mptcp11.png

Once the connection has been established, the server sends quickly a packet that advertises its own address in the ADD_ADDR option.

../../../_images/mptcp21.png

A closer look at this option indicates that this address is advertised as having address identifier 0. According to RFC6824, this identifier is reserved for the address of the initial subflow. This advertisement thus changes the address of the initial subflow, and we can expect that iOS13 will use this new address to establish subsequent subflows on this Multipath TCP connection. We checked by contacting one of Apple Music servers from a Linux client to see how Linux reacts to such an option and it supports it correctly.

../../../_images/mptcp3.png

According to statista, there were more than 60 million subscribers of the Apple Music service in June 2019. As they upgrade their smartphones to support iOS13, we will observe a huge growth in Multipath TCP traffic during the coming weeks… If you see other servers or CDNs that enable Multipath TCP, let us know...

SOCKS and Multipath TCP

The main benefit of Multipath TCP is that it enables either the simultaneous utilisation of different paths or provides fast handovers from one path to another. Several deployments of Multipath TCP leverage these unique capabilities of multipath transport. When Apple opted for Multipath TCP in 2013, they could leverage its benefits by enabling it on both their iPhones and iPads and on the servers that support the Siri application. This is the end-to-end approach to deploying Multipath TCP.

However, there are many deployment scenarios where it would be beneficial to use Multipath TCP to interact with servers that have not yet been upgraded to support this new protocol. A first use case are the smartphones willing to combine Wi-Fi and cellular or quickly switch from one to another. Another use case is to combine different access networks to benefit from a higher bandwidth. In large cities, users can often use high bandwidth access links through VDSL, FFTH or cable networks. However, in rural areas, bandwidth is often limited and many home users or SMEs need to combine different access links to reach 10 Mbps or a small multiple of this low bandwidth. Multipath TCP has been used in these two deployments together with SOCKS RFC 1928.

SOCKS is an old application layer protocol that was designed to allow users in a corporate network to access the Internet through a firewall that filters connections. Several implementations of SOCKS exist and a popular one is shadowsocks. In Soutch Korea, the smarpthones that use the Gigapath service interact through a SOCKS server over Multipath TCP as illustrated in the figure below.

../../../_images/KT-fig1.png

This SOCKS service can be easily deployed once the Multipath TCP implementation has been ported on the smartphones. It appears as a simple application that interacts with the SOCKS server which is managed by the network operator. However, SOCKS is a chatty protocol that significantly increases the delay to establish a connection. To establish one TCP connection through the SOCKS proxy, the smartphone needs to exchange several packets as shown in the figure below.

../../../_images/socks-mptcp.png

First, the smartphone sends a SYN with the MP_CAPABLE option to create a connection to the SOCKS server. The SOCKS server replies and this consumes one round-trip-time. Then, the smartphone sends a SOCKS message to initiate the SOCKS sessions and find the methods that the SOCKS server supports. In some cases, an additional step is required to authenticate the client (not shown in the figure). Finally, the smartphone sends a CONNECT message to request the creation of a connection towards the final server. At this point, the smartphone can also create a subflow towards the SOCKS server. A benefit of some SOCKS deployment is that it can also encrypt all the data exchanged between the smartphone and the SOCKS server. This was important a few years ago, but less crucial today given the number of services that already use TLS.

A similar approach has been used to combine different access links. For example, OVH provides the overthebox that uses a SOCKS proxy running on an access router to combine several xDSL/cable/cellular links in a single pipe. The SOCKS proxy running on this router interacts with a SOCKS server as shown above. The code running on their access router is open-source and available from https://github.com/ovh/overthebox. OpenMPTCPRouter uses a similar approach.

The main benefit of SOCKS in these deployment is that it enables the simultaneous utilization of different access links. This increases the throughput for long TCP connections as shown by measurements from speedtest and similar services. However, SOCKS has a major drawback: it increases the time required to establish all TCP connections by several round-trip-times between the client and the SOCKS server. This additional delay can be significant for applications that rely on short TCP connections. The figure below (source [CBHB16a]) shows that the round-trip-time on cellular networks can be significant.

../../../_images/socks-rtt.png

Measurement carried on smartphones [CBHB16a] show that many applications use very short connections that exchange a small amount of data. Increasing the setup time of these connections by forcing them to be proxied by SOCKS may affect the user experience.

../../../_images/socks-measurements.png

References

[CBHB16a](1, 2) Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. A first analysis of multipath tcp on smartphones. In 17th International Passive and Active Measurements Conference, volume 17. Springer, March-April 2016. URL: https://inl.info.ucl.ac.be/publications/first-analysis-multipath-tcp-smartphones.html.

Using load balancers in front of Multipath TCP servers

Load balancers play a very important role in today’s Internet. Most Internet services are provided by servers that reside behind one or several layers of load-balancers. Various load-balancers have been proposed and implemented. They can operate at layer 3, layer 4 or layer 7. Layer 4 is very popular and we focus on such load balancers in this blog post. A layer-4 load balancer uses information from the transport layer to load balance TCP connections over different servers. There are two main types of layer-4 load balancers :

  • The stafeful load balancers
  • The stateless load balancers

Schematically, a load balancer is a device or network function that processes incoming packets and forwards all packets that belong to the same connection to a specific server. A stateful load balancer will maintain a table that associates the five-tuple that identifies a TCP connection to a specific server. When a packet arrives, it seeks a matching entry in the table. If a match is found, the packet is forwarded to the selected server. If there is no match, e.g. the packet is a SYN, a server is chosen and the table is updated before forwarding the packet. The table entries are removed when they expire or when the associated connection is closed. A stateless load balancer does not maintain a table. Instead, it relies on hash function that is computed over each incoming packet. A simple approach is to use a CRC over the source and destination addresses and ports and associate each server to a range of CRC values. This is illustrated in the figure below (source: Fabien Duchene’s presentation of [DB17]).

../../../_images/lb-1.png

With Multipath TCP, a single connection can be composed of different subflows that have their own five tuples. This implies that that data corresponding to a given Multipath TCP connection can be received over several different TCP subflows that obviously need to be forwarded to the same server by the load balancer. This is illustrated in the figure below (source: Fabien Duchene’s presentation of [DB17]).

../../../_images/lb-2.png

Commercial load balancers

Several approaches have been proposed in the literature to solve this problem. In Datacenter Scale Load Balancing for Multipath Transport [OR16], V. Olteanu and C. Raiciu proposed two different tricks to support stateless load balancers with Multipath TCP. First, the load balancer selects the key that will be used by the server for each incoming Multipath TCP connection. As this key is used to Token that identifies the Multipath connection in the MP_JOIN option, this enables the load balancer to control the Token that clients will send when creating subflows. This allows the load balancer to correctly associated MP_JOINs to the server that terminates the corresponding connection. This is not sufficient for a stateless load balancer. A stateless load balancer also needs to associate each incoming packet to a specific server. If this packet belongs to a subflow, it carries the source and destination addresses and ports, but those of a subflow have no relationship with the initial subflow. They solve this problem by encoding the identification of the server inside a part of the TCP timestamp option.

In Towards a Multipath TCP Aware Load Balancer [LienardyD16], S. Lienardy and B. Donnet propose a mix between stateless and stateful approaches. The packets from the first subflow are sent to a specific server by hashing their source and destination addresses and ports. They then extract the key exchanged in the third ack to store the token associated with this connection. This token is then placed in a map that is used to load balance the SYN MP_JOIN packets. The reception of an MP_JOIN packet forces the creation of an entry in a table that is used to map the packets from the additional subflows. This is illustrated in the figure below (source [LienardyD16])

../../../_images/lb-3.png

A benefit of this approach is that since the second subflow is directly associated with the server, all the packets exchanged over this subflow can reach the server without passing through the load balancer.

../../../_images/lb-4.png

In Making Multipath TCP friendlier to Load Balancers and Anycast, F. Duchene and O. Bonaventure leverage a feature of the forthcoming standard’s track version of Multipath TCP. In this revision, the MP_CAPABLE option has been modified compared to RFC6824. A first modification is that the client does not send its key anymore in the SYN packet. A second modification is the C that when when set by a server in the SYN+ACK, it indicates that the server will not accept additional MPTCP subflows to the source address and flows of the SYN. This bit was specifically introduced to support load balancers. It works as follows. When a client creates a connection, it sends a SYN towards the load balancer with the MP_CAPABLE option but no key. The load balancer selects one server to handle the connection, e.g. based on a stateless hash. Each server has a dedicated IP address or a dedicated port number. It replies to the SYN with a SYN+ACK that contains the MP_CAPABLE option with the C bit set. Once the connection is established, it sends an ADD_ADDR option with its direct IP address to the client. The client then uses the direct address to create the subflows and those can completely bypass the load balancer. The source code of the implementation is available from https://github.com/fduchene/ICNP2017

The latest Multipath TCP load balancer was proposed in Stateless Datacenter Load-balancing with Beamer by V. Olteanu et al. It assigns one port to each load balanced server and also forces the client to create the subflows towards this per-server port number. The load balancer is implemented in both software (click elements) and hardware (P4) and evaluated in details. The source code is available from https://github.com/Beamer-LB

Commercial load balancers such as F5 or Citrix also support Multipath TCP.

References

[DB17](1, 2) Fabien Duchene and Olivier Bonaventure. Making multipath tcp friendlier to load balancers and anycast. In Network Protocols (ICNP), 2017 IEEE 25th International Conference on, 1–10. IEEE, 2017. URL: https://inl.info.ucl.ac.be/publications/making-multipath-tcp-friendlier-load-balancers-and-anycast.html.
[LienardyD16](1, 2) Simon Liénardy and Benoit Donnet. Towards a multipath tcp aware load balancer. In Proceedings of the 2016 Applied Networking Research Workshop, 13–15. ACM, 2016.
[OR16]Vladimir Olteanu and Costin Raiciu. Datacenter scale load balancing for multipath transport. In Proceedings of the 2016 workshop on Hot topics in Middleboxes and Network Function Virtualization, 20–25. ACM, 2016.

Which servers use Multipath TCP ?

Since the publication of RFC 6824, Multipath TCP has been deployed for several use cases described in RFC 8041 and [BS16]. However, there has been no public study of the deployment of Multipath TCP on servers.

A first study appeared in [MHFB15]. They used zmap to send SYN packets with the MP_CAPABLE option and checked whether they replied with the MP_CAPABLE option. They used 452,008 unique IP addresses corresponding to a bit less than 2 million domains. Their results are summarised in the table below (source [MHFB15]).

../../../_images/mptcp-scan.png

Unfortunately, there was a flaw in the methodology. Checking the presence of the MP_CAPABLE option in the SYN+ACKis not sufficient as some middleboxes simply echo in the SYN+ACK any option contained in the SYN even if they do not understand it. The dataset was later corrected and updated. Any zmap scan must check that a returned TCP option in a SYN+ACK is not simply echoed by a middlebox. For Multipath TCP, this means that the returned MP_CAPABLE option must differ from the option sent in the SYN.

Since the publication of this paper, no other study using zmap has been published. Several research groups use zmap to carry out measurement studies. One of them is the netray Internet Observatory at RWTH Aachen University. They have used zmap to study the deployment of HTTP/2, QUIC or the initial TCP congestion window. I recently discussed with Jan Ruth and Oliver Hohlfeld who had some interesting data. In August 2015, a scan of the entire IPv4 addressing space revealed 921 Multipath TCP servers, with about one sixth of them hosted by Apple. At the same time, about 24.000 addresses returned in the SYN+ACK an MP_CAPABLE option that was exactly the same as the one sent in the SYN packet.

They proposed to launch another zmap scan last week over the entire IPv4 addressing space on port 443. 88484 addresses replied with an MP_CAPABLE option. Out of these, 77384 simply returned exactly the same MP_CAPABLE option as the one included in the SYN packet. There are thus more middleboxes that echo unsupported TCP options than in August 2015. The interesting point of their study is that 11k addresses replied with a valid MP_CAPABLE option. This indicates an important growth in the deployment of Multipath TCP on servers. This analysis ignores the utilisation of Multipath TCP on clients such as iPhones and other smartphones, but also in Hybrid Access Networks since none of those devices respond to SYN packets.

The exact usage of these servers is not known. If you manage one of them, we’d be interested in knowing more about your use case to document it in revisions of RFC 8041. If it receives a lot of traffic, we’d also be interested in analysing packet traces in more details to extend the measurements published in [CBHB16a][CBHB16b][HTSB15][TCH+16].

References

[BS16]Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL: https://www.ietfjournal.org/multipath-tcp-deployments/.
[CBHB16a]Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. A first analysis of multipath tcp on smartphones. In 17th International Passive and Active Measurements Conference, volume 17. Springer, March-April 2016. URL: https://inl.info.ucl.ac.be/publications/first-analysis-multipath-tcp-smartphones.html.
[CBHB16b]Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. Observing real smartphone applications over multipath tcp. IEEE Communications Magazine, Network Testing Series, March 2016. URL: https://inl.info.ucl.ac.be/publications/observing-real-smartphone-applications-over-multipath-tcp.html.
[HTSB15]Benjamin Hesmans, Viet-Hoang Tran, Ramin Sadre, and Olivier Bonaventure. A first look at real multipath tcp traffic. In Traffic Monitoring and Analysis. 2015. URL: https://inl.info.ucl.ac.be/publications/first-look-real-multipath-tcp-traffic.html.
[MHFB15](1, 2) Olivier Mehani, Ralph Holz, Simone Ferlin, and Roksana Boreli. An early look at multipath tcp deployment in the wild. In Proceedings of the 6th International Workshop on Hot Topics in Planet-Scale Measurement, HotPlanet ‘15, 7–12. New York, NY, USA, 2015. ACM. URL: http://doi.acm.org/10.1145/2798087.2798088, doi:10.1145/2798087.2798088.
[TCH+16]Viet-Hoang Tran, Quentin De Coninck, Benjamin Hesmans, Ramin Sadre, and Olivier Bonaventure. Observing real multipath tcp traffic. Computer Communications, 2016. URL: https://inl.info.ucl.ac.be/publications/observing-real-multipath-tcp-traffic.html, doi:10.1016/j.comcom.2016.01.014.