Simplifying Multipath TCP configuration with Babel

Multipath TCP builds its sub-flows based on pairs of the client’s and server’s IP addresses. When a host is connected to two different providers, it should have one IP address associated to each provider, which allows Multipath TCP to effectively use these two paths simultaneously. However, with a single default route, packets will follow the same path, independently of their source address, which prevents Multipath TCP from working properly.

The Multipath TCP website provides a recipe for configuring multiple routes manually on a single host directly connected to multiple providers using policy routing. Even for such simple topology, the procedure consists of 7 commands. Analogous configuration for a more complex network, while possible, is exceedingly tedious, error-prone and fragile.

Consider for example the following topology, where routers A and B are connected to distinct providers, and the host H is connected to router C. Routers A and B have default routes for packets sourced in 10.1.1.0/24 and 10.1.2.0/24 respectively.

     0.0.0.0/0    --- A ----- B ---    0.0.0.0/0
from 10.1.1.0/24       \     /      from 10.1.2.0/24
                        \   /
                         \ /
                          C
                          |
                          |
                          H

A manual configuration of this network will require an intervention on at least routers A, B and C. If for some reason the topology changes, a link fails or a router crashes, the network will fail to work correctly until an explicit intervention of the administrator.

Routing configuration of complex networks should not be done manually. That is the role of a dynamic routing protocol.

The source-specific version of the Babel routing protocol RFC 6126 will dynamically configure a network to route packets based both on both their source and destination addresses. Babel will learn routes installed in the kernel of the gateway routers (A and B), announce them through the network, and install them in the routing tables of the routers. Only the edge gateways need manual configuration to teach Babel which source prefix to apply to the default route they provide. A contrario, internal routers (C) will not need any manual configuration. In our example, we need to add the following directives to the Babel configuration file (/etc/babeld.conf) of the two gateways:

../../../_images/babel.png

On router A:

redistribute ip 0.0.0.0/0 eq 0 src-prefix 10.1.1.0/24
redistribute ip 0.0.0.0/0 eq 0 proto 3 src-prefix 10.1.1.0/24

On router B:

redistribute ip 0.0.0.0/0 eq 0 src-prefix 10.1.2.0/24
redistribute ip 0.0.0.0/0 eq 0 proto 3 src-prefix 10.1.2.0/24

The second line specifies proto 3 because Babel does not redistribute proto boot routes by default.

The case of IPv6 is similar. In fact, it may be simpler, because Babel recognises kernel-side source-specific routes: in the best case, no additional configuration may be needed in IPv6.

The source-specific version of Babel works on Linux, and is available in OpenWRT as the babels package. You can also retrieve its source code from:

git clone https://github.com/boutier/babeld

Additional information about the Babel protocol and its operation with Multipath TCP may be found in the technical report entitled Source-specific routing written by Matthieu Boutier and Juliusz Chroboczek

Useful Multipath TCP software

Since the first release of the Multipath TCP implementation in the Linux kernel, various software packages have been written by project students and project contributors. Some of this software is available from the official Multipath TCP github repository. The core packages are :

In addition to these packages, the repository contains several ports for specific platforms including the Raspberry PI and several smartphones. However, most of these ports were based on an old version of the Multipath TCP kernel and have not been maintained.

Extensions to tcpdump and wireshark were initially posted as patches. The basic support for Multipath TCP is now included in these two important packages. Extensions are still being developed, e.g. https://github.com/teto/wireshark

Besides these core packages, several other open-source packages can be very useful for Multipath TCP users and developers :

  • tracebox is a flexible traceroute-like package that allows to detect middlebox interference inside a network. It can be used to verify whether packets containing any of the MPTCP options pass through firewalls. The source code for tracebox is available from https://github.com/tracebox/tracebox
  • mptcp-scapy is an extension to the popular scapy packet manipulation software that supports Multipath TCP. It was recently updated and can be obtained from https://github.com/Neohapsis/mptcp-abuse
  • MPTCPtrace is the equivalent for Multipath TCP of the popular tcptrace packages. It parses libpcap files and extracts both statistics and plots from the packet capture. This is one of the best way to analyse the dynamics of Multipath TCP in a real network.
  • MBClick is a set of Click elements that reproduce the behaviour of various types of middleboxes. It proved to be very useful while developing the middlebox support in Multipath TCP.
  • packetdrill is a software that allows to test precisely the behaviour of a TCP implementation. It has been extended to support Multipath TCP and a set of tests has been developed : https://github.com/ArnaudSchils/packetdrill_mptcp
  • MPTCP vagrant is a set of scripts that can be used to automate the creation of Multipath TCP capable virtualbox images.

A first look at multipath-tcp.org : subflows

In theory, a Multipath TCP connection can gather an unlimited number of subflows. In practice, implementations limit the number of concurrent subflows. The Linux implementation used on the monitored server can support up to 32 different subflows. We analyse here the number of subflows that are established for each Multipath TCP connection. Since our server never establishes subflows, this number is an indication of the capabilities of the clients that interact with it.

The figure below provides the distribution of the number of subflows per Multipath TCP connection. We show the distribution for the number of successfully established subflows, i.e., subflows that complete the handshake, as well as for all attempted ones. As can be seen, several connection attempts either fail completely or establish less subflows than intended. In total, we observe 5098 successful connections with 8701 subflows. The majority of the observed connections (57%) only establish one subflow. Around 27% of them use two subflows. Only 10 connections use more than 8 subflows, which are omitted from the figure.

../../../_images/nflows.png

A first look at multipath-tcp.org : ADD_ADDR usage

The first question that we asked ourselves about the usage of Multipath TCP was whether the communicating hosts were using multiple addresses.

Since the packet trace was collected on the server that hosts the Multipath TCP implementation in the Linux kernel, we can expect that many Linux enthusiasts use it to download new versions of the code, visit the documentation, perform tests or verify that their configuration is correct. These users might run different versions of Multipath TCP in the Linux kernel or on other operating systems. Unfortunately, as of this writing, there is not enough experience with the Multipath TCP implementations to detect which operating system was used to generate specific Multipath TCP packets.

Thanks to the ADD_ADDR option, it is however possible to collect interesting data about the characteristics of the clients that contact our server. Over the 5098 observed Multipath TCP connections, 3321 of them announced at least one address. Surprisingly, only 21% of the collected IPv4 addresses in the ADD_ADDR option were globally routable address.

The remaining 79% of the IPv4 addresses found in the ADD_ADDR option were private addresses and in some cases link-local addresses. This confirms that Multipath TCP’s ability to pass through NATs is an important feature of the protocol RFC 6824.

The IPv6 addresses collected in the ADD_ADDR option had more diversity. We first observed 72% of globally routable IPv6 addresses. The other types of addresses that we observed are shown in the table below. The IPv4-compatible and the 6to4 IPv6 addresses were expected, but the link local and documentation addresses should have been filtered by the client and not be announced over Multipath TCP connections. The Multipath TCP specification RFC 6824 should be updated to specify which types of IPv4 and IPv6 addresses can be advertised over a Multipath TCP connection.

Address type Count
Link-local (IPv4) 51
Link-local (IPv6) 241
Documentation only (IPv6) 21
IPv4-compatible IPv6 13
6to4 206