SOCKS and Multipath TCP

The main benefit of Multipath TCP is that it enables either the simultaneous utilisation of different paths or provides fast handovers from one path to another. Several deployments of Multipath TCP leverage these unique capabilities of multipath transport. When Apple opted for Multipath TCP in 2013, they could leverage its benefits by enabling it on both their iPhones and iPads and on the servers that support the Siri application. This is the end-to-end approach to deploying Multipath TCP.

However, there are many deployment scenarios where it would be beneficial to use Multipath TCP to interact with servers that have not yet been upgraded to support this new protocol. A first use case are the smartphones willing to combine Wi-Fi and cellular or quickly switch from one to another. Another use case is to combine different access networks to benefit from a higher bandwidth. In large cities, users can often use high bandwidth access links through VDSL, FFTH or cable networks. However, in rural areas, bandwidth is often limited and many home users or SMEs need to combine different access links to reach 10 Mbps or a small multiple of this low bandwidth. Multipath TCP has been used in these two deployments together with SOCKS RFC 1928.

SOCKS is an old application layer protocol that was designed to allow users in a corporate network to access the Internet through a firewall that filters connections. Several implementations of SOCKS exist and a popular one is shadowsocks. In Soutch Korea, the smarpthones that use the Gigapath service interact through a SOCKS server over Multipath TCP as illustrated in the figure below.


This SOCKS service can be easily deployed once the Multipath TCP implementation has been ported on the smartphones. It appears as a simple application that interacts with the SOCKS server which is managed by the network operator. However, SOCKS is a chatty protocol that significantly increases the delay to establish a connection. To establish one TCP connection through the SOCKS proxy, the smartphone needs to exchange several packets as shown in the figure below.


First, the smartphone sends a SYN with the MP_CAPABLE option to create a connection to the SOCKS server. The SOCKS server replies and this consumes one round-trip-time. Then, the smartphone sends a SOCKS message to initiate the SOCKS sessions and find the methods that the SOCKS server supports. In some cases, an additional step is required to authenticate the client (not shown in the figure). Finally, the smartphone sends a CONNECT message to request the creation of a connection towards the final server. At this point, the smartphone can also create a subflow towards the SOCKS server. A benefit of some SOCKS deployment is that it can also encrypt all the data exchanged between the smartphone and the SOCKS server. This was important a few years ago, but less crucial today given the number of services that already use TLS.

A similar approach has been used to combine different access links. For example, OVH provides the overthebox that uses a SOCKS proxy running on an access router to combine several xDSL/cable/cellular links in a single pipe. The SOCKS proxy running on this router interacts with a SOCKS server as shown above. The code running on their access router is open-source and available from OpenMPTCPRouter uses a similar approach.

The main benefit of SOCKS in these deployment is that it enables the simultaneous utilization of different access links. This increases the throughput for long TCP connections as shown by measurements from speedtest and similar services. However, SOCKS has a major drawback: it increases the time required to establish all TCP connections by several round-trip-times between the client and the SOCKS server. This additional delay can be significant for applications that rely on short TCP connections. The figure below (source [CBHB16a]) shows that the round-trip-time on cellular networks can be significant.


Measurement carried on smartphones [CBHB16a] show that many applications use very short connections that exchange a small amount of data. Increasing the setup time of these connections by forcing them to be proxied by SOCKS may affect the user experience.



[CBHB16a](1, 2) Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. A first analysis of multipath tcp on smartphones. In 17th International Passive and Active Measurements Conference, volume 17. Springer, March-April 2016. URL:

Using load balancers in front of Multipath TCP servers

Load balancers play a very important role in today’s Internet. Most Internet services are provided by servers that reside behind one or several layers of load-balancers. Various load-balancers have been proposed and implemented. They can operate at layer 3, layer 4 or layer 7. Layer 4 is very popular and we focus on such load balancers in this blog post. A layer-4 load balancer uses information from the transport layer to load balance TCP connections over different servers. There are two main types of layer-4 load balancers :

  • The stafeful load balancers
  • The stateless load balancers

Schematically, a load balancer is a device or network function that processes incoming packets and forwards all packets that belong to the same connection to a specific server. A stateful load balancer will maintain a table that associates the five-tuple that identifies a TCP connection to a specific server. When a packet arrives, it seeks a matching entry in the table. If a match is found, the packet is forwarded to the selected server. If there is no match, e.g. the packet is a SYN, a server is chosen and the table is updated before forwarding the packet. The table entries are removed when they expire or when the associated connection is closed. A stateless load balancer does not maintain a table. Instead, it relies on hash function that is computed over each incoming packet. A simple approach is to use a CRC over the source and destination addresses and ports and associate each server to a range of CRC values. This is illustrated in the figure below (source: Fabien Duchene’s presentation of [DB17]).


With Multipath TCP, a single connection can be composed of different subflows that have their own five tuples. This implies that that data corresponding to a given Multipath TCP connection can be received over several different TCP subflows that obviously need to be forwarded to the same server by the load balancer. This is illustrated in the figure below (source: Fabien Duchene’s presentation of [DB17]).


Commercial load balancers

Several approaches have been proposed in the literature to solve this problem. In Datacenter Scale Load Balancing for Multipath Transport [OR16], V. Olteanu and C. Raiciu proposed two different tricks to support stateless load balancers with Multipath TCP. First, the load balancer selects the key that will be used by the server for each incoming Multipath TCP connection. As this key is used to Token that identifies the Multipath connection in the MP_JOIN option, this enables the load balancer to control the Token that clients will send when creating subflows. This allows the load balancer to correctly associated MP_JOINs to the server that terminates the corresponding connection. This is not sufficient for a stateless load balancer. A stateless load balancer also needs to associate each incoming packet to a specific server. If this packet belongs to a subflow, it carries the source and destination addresses and ports, but those of a subflow have no relationship with the initial subflow. They solve this problem by encoding the identification of the server inside a part of the TCP timestamp option.

In Towards a Multipath TCP Aware Load Balancer [LienardyD16], S. Lienardy and B. Donnet propose a mix between stateless and stateful approaches. The packets from the first subflow are sent to a specific server by hashing their source and destination addresses and ports. They then extract the key exchanged in the third ack to store the token associated with this connection. This token is then placed in a map that is used to load balance the SYN MP_JOIN packets. The reception of an MP_JOIN packet forces the creation of an entry in a table that is used to map the packets from the additional subflows. This is illustrated in the figure below (source [LienardyD16])


A benefit of this approach is that since the second subflow is directly associated with the server, all the packets exchanged over this subflow can reach the server without passing through the load balancer.


In Making Multipath TCP friendlier to Load Balancers and Anycast, F. Duchene and O. Bonaventure leverage a feature of the forthcoming standard’s track version of Multipath TCP. In this revision, the MP_CAPABLE option has been modified compared to RFC6824. A first modification is that the client does not send its key anymore in the SYN packet. A second modification is the C that when when set by a server in the SYN+ACK, it indicates that the server will not accept additional MPTCP subflows to the source address and flows of the SYN. This bit was specifically introduced to support load balancers. It works as follows. When a client creates a connection, it sends a SYN towards the load balancer with the MP_CAPABLE option but no key. The load balancer selects one server to handle the connection, e.g. based on a stateless hash. Each server has a dedicated IP address or a dedicated port number. It replies to the SYN with a SYN+ACK that contains the MP_CAPABLE option with the C bit set. Once the connection is established, it sends an ADD_ADDR option with its direct IP address to the client. The client then uses the direct address to create the subflows and those can completely bypass the load balancer. The source code of the implementation is available from

The latest Multipath TCP load balancer was proposed in Stateless Datacenter Load-balancing with Beamer by V. Olteanu et al. It assigns one port to each load balanced server and also forces the client to create the subflows towards this per-server port number. The load balancer is implemented in both software (click elements) and hardware (P4) and evaluated in details. The source code is available from

Commercial load balancers such as F5 or Citrix also support Multipath TCP.


[DB17](1, 2) Fabien Duchene and Olivier Bonaventure. Making multipath tcp friendlier to load balancers and anycast. In Network Protocols (ICNP), 2017 IEEE 25th International Conference on, 1–10. IEEE, 2017. URL:
[LienardyD16](1, 2) Simon Liénardy and Benoit Donnet. Towards a multipath tcp aware load balancer. In Proceedings of the 2016 Applied Networking Research Workshop, 13–15. ACM, 2016.
[OR16]Vladimir Olteanu and Costin Raiciu. Datacenter scale load balancing for multipath transport. In Proceedings of the 2016 workshop on Hot topics in Middleboxes and Network Function Virtualization, 20–25. ACM, 2016.

Which servers use Multipath TCP ?

Since the publication of RFC 6824, Multipath TCP has been deployed for several use cases described in RFC 8041 and [BS16]. However, there has been no public study of the deployment of Multipath TCP on servers.

A first study appeared in [MHFB15]. They used zmap to send SYN packets with the MP_CAPABLE option and checked whether they replied with the MP_CAPABLE option. They used 452,008 unique IP addresses corresponding to a bit less than 2 million domains. Their results are summarised in the table below (source [MHFB15]).


Unfortunately, there was a flaw in the methodology. Checking the presence of the MP_CAPABLE option in the SYN+ACKis not sufficient as some middleboxes simply echo in the SYN+ACK any option contained in the SYN even if they do not understand it. The dataset was later corrected and updated. Any zmap scan must check that a returned TCP option in a SYN+ACK is not simply echoed by a middlebox. For Multipath TCP, this means that the returned MP_CAPABLE option must differ from the option sent in the SYN.

Since the publication of this paper, no other study using zmap has been published. Several research groups use zmap to carry out measurement studies. One of them is the netray Internet Observatory at RWTH Aachen University. They have used zmap to study the deployment of HTTP/2, QUIC or the initial TCP congestion window. I recently discussed with Jan Ruth and Oliver Hohlfeld who had some interesting data. In August 2015, a scan of the entire IPv4 addressing space revealed 921 Multipath TCP servers, with about one sixth of them hosted by Apple. At the same time, about 24.000 addresses returned in the SYN+ACK an MP_CAPABLE option that was exactly the same as the one sent in the SYN packet.

They proposed to launch another zmap scan last week over the entire IPv4 addressing space on port 443. 88484 addresses replied with an MP_CAPABLE option. Out of these, 77384 simply returned exactly the same MP_CAPABLE option as the one included in the SYN packet. There are thus more middleboxes that echo unsupported TCP options than in August 2015. The interesting point of their study is that 11k addresses replied with a valid MP_CAPABLE option. This indicates an important growth in the deployment of Multipath TCP on servers. This analysis ignores the utilisation of Multipath TCP on clients such as iPhones and other smartphones, but also in Hybrid Access Networks since none of those devices respond to SYN packets.

The exact usage of these servers is not known. If you manage one of them, we’d be interested in knowing more about your use case to document it in revisions of RFC 8041. If it receives a lot of traffic, we’d also be interested in analysing packet traces in more details to extend the measurements published in [CBHB16a][CBHB16b][HTSB15][TCH+16].


[BS16]Olivier Bonaventure and SungHoon Seo. Multipath tcp deployments. IETF Journal, 12(2):24–27, 2016. URL:
[CBHB16a]Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. A first analysis of multipath tcp on smartphones. In 17th International Passive and Active Measurements Conference, volume 17. Springer, March-April 2016. URL:
[CBHB16b]Quentin De Coninck, Matthieu Baerts, Benjamin Hesmans, and Olivier Bonaventure. Observing real smartphone applications over multipath tcp. IEEE Communications Magazine, Network Testing Series, March 2016. URL:
[HTSB15]Benjamin Hesmans, Viet-Hoang Tran, Ramin Sadre, and Olivier Bonaventure. A first look at real multipath tcp traffic. In Traffic Monitoring and Analysis. 2015. URL:
[MHFB15](1, 2) Olivier Mehani, Ralph Holz, Simone Ferlin, and Roksana Boreli. An early look at multipath tcp deployment in the wild. In Proceedings of the 6th International Workshop on Hot Topics in Planet-Scale Measurement, HotPlanet ‘15, 7–12. New York, NY, USA, 2015. ACM. URL:, doi:10.1145/2798087.2798088.
[TCH+16]Viet-Hoang Tran, Quentin De Coninck, Benjamin Hesmans, Ramin Sadre, and Olivier Bonaventure. Observing real multipath tcp traffic. Computer Communications, 2016. URL:, doi:10.1016/j.comcom.2016.01.014.

Advertising addresses with Multipath TCP

An important feature of Multipath TCP is that the communicating hosts can easily learn the addresses that can be used to reach their peers. Multipath TCP uses special TCP options to advertise this information. In this post, we look at the evolution of these options during the design of RFC 6824

The first MPTCP draft, draft-ford-mptcp-multiaddressed-00 defined four options to deal with addresses.


The first option was called Add Address. It was intended to advertise one IP address owned by the host that sends it over a given Multipath TCP connection. This option contained an IP version field, to support IPv4 and IPv6, and an index. This index was intended to cope with NAT that could translate addresses in the IP header without translating information in the TCP options.


A companion option was the remove address option that simply contained the index of the address to be removed.


In addition to these two options that have been slightly modified later in RFC 6824, draft-ford-mptcp-multiaddressed-00 also defined two options for implicit path management. The first one is Request-SYN. It was intended to request the peer to initiate a subflow towards the announced address. The Add Address option was intended to be purely informational and the Request-SYN was intended to trigger the establishment of a new subflow.


There was also a Request-FIN option that was intended to request the termination of a subflow.


These two options were removed in draft-ford-mptcp-multiaddressed-01 and only the Add and Remove address options were kept.

The working group draft, draft-ietf-mptcp-multiaddressed-00 introduced another two changes to these options. First, it became possible to advertise a specific port number together with an IP address. Second, the document included a discussion on adding a backup flag to indicate that the advertised address should be treated as a backup one. This bit became fully defined in draft-ietf-mptcp-multiaddressed-02. As of December 2018, it is unclear whether the ability to advertise a port number is used by a real implementation. Some measurements about the utilization of this option on are discussed in [HTSB15] and [TCH+16].


Then, the backup bit was removed in draft-ietf-mptcp-multiaddressed-03. In draft-ietf-mptcp-multiaddressed-04, a new option called MP_PRIO was introduced to allow a host to dynamically change the priority (backup or not) of an address. This separation between the address advertisement and the announcement of the backup status was intended to provide more reactivity.


This option is sent by a receiver to indicate to its peer that it wants to change the backup status of the subflow over which this option is sent.

It should be noted that the MP_JOIN option that is used to create subflows contains a backup flag that allows to indicate whether a new subflow should be treated as a backup one or not. In contrast with the MP_PRIO option that is sent unreliably in a TCP option, the MP_JOIN option is always exchanged reliably, which guarantees that the communicating hosts know the backup status of newly established flows.

The discussion on advertising addresses continued within the IETF after the publication of RFC 6824. The Linux implementation supports this option and uses it to announce a new address as soon as this address is known. However, the reception of an ADD_ADDR option does not necessarily trigger the establishment of a subflow. This is under the responsibility of the path manager. The fullmesh patch manager running on a client tries to use all possible addresses, but a Linux server does not create subflows. Apple’s implementation does not support the ADD_ADDR option. They assume that the servers have a single IP address and that the client will anyway create subflows by using the MP_JOIN option.

One of the concerns with the ADD_ADDR option included in RFC 6824 was the risk of attacks where an attacker could inject his address in an existing Multipath TCP connection by sending a spoofed packet containing this option and valid TCP sequence numbers. This attack was described in the threats analysis RFC 7430. The new ADD_ADDR option was proposed at IETF88 and included in draft-ietf-mptcp-rfc6824bis-00. It includes a truncated HMAC that authenticates it with the keys exchanged during the initial handshake. It also ensures that this option is not modified by an on-path attacker that has not observed the initial handshake.


The discussion continued and draft-ietf-mptcp-rfc6824bis-01 introduced an Echo bit to provide a reliable delivery of this option.


As of December 2018, this option is included in the last version of draft-ietf-mptcp-rfc6824bis-12 but has not yet been deployed in the field.


[HTSB15]Benjamin Hesmans, Viet-Hoang Tran, Ramin Sadre, and Olivier Bonaventure. A first look at real multipath tcp traffic. In Traffic Monitoring and Analysis. 2015. URL:
[TCH+16]Viet-Hoang Tran, Quentin De Coninck, Benjamin Hesmans, Ramin Sadre, and Olivier Bonaventure. Observing real multipath tcp traffic. Computer Communications, 2016. URL:, doi:10.1016/j.comcom.2016.01.014.