A common problem for evaluating multiple transport protocols in a multi-platform environment is to have a test tool that is capable to run in all these environments, and – of course – to support all necessary protocols. Using different evaluation tools is not a good solution, since each tool may introduce its own – and possibly incompatible – parametrisation scheme. In order to overcome this problem, originally for the use case of evaluating the Stream Control Transmission Protocol (SCTP) and compare it to the Transmission Control Protocol (TCP), NetPerfMeter has been designed and developed.
What is NetPerfMeter?
NetPerfMeter  is an open source, multi-platform transport protocol performance evaluation software. It currently supports the Linux, FreeBSD and MacOS platforms (with possibility to easily extend it to further platforms), and the transport protocols SCTP, TCP including Multi-Path TCP (MPTCP, if supported by the operating system), UDP (User Datagram Protocol) and DCCP (Datagram Congestion Control Protocol, if supported by the operating system). The figure below presents the NetPerfMeter protocol stack.
In each direction, a data channel can be operated in saturated mode (send as much as possible; not available for UDP-based data channels) or non-saturated mode. In case of the non-saturated mode, the traffic is configured in form of frame rates and frame sizes. That is, frames of a given size are generated in given intervals. Both, frame rate and frame size can be randomised as well.
The performance of the data channels (bandwidth, delay, jitter, etc.) is evaluated and recorded at active and passive side. At the end of a measurement, all collected data is transferred over the control channel to the active side. That is, a user can just conveniently collect all results at the active side. Further details on NerPerfMeter can be found in . ,  also introduce an OMNeT++ simulation model for NetPerfMeter that can be used in simulations, in order to easily compare simulation results and real-world measurements. ,  and  provide examples of NetPerfMeter usage for SCTP and MPTCP protocol performance analyses.
Testing MPTCP with NetPerfMeter
In the following, a short tutorial for using NetPerfMeter with MPTCP is provided. It explains the basic features to start experimenting. A more detailed overview of all possible options can be found in the manpage of NetPerfMeter.
Starting the Passive Side
First, the passive side of NetPerfMeter needs to be started:
This command starts NetPerfMeter in server mode, waiting for control channel connections on port 9000.
Starting the Active Side
One Non-Saturated TCP Flow
Let us start with just one TCP flow:
netperfmeter <passive side>:9000 -tcp const10:const1460:const0:const0
This command starts NetPerfMeter in client mode, connecting to the given peer (given by hostname or IP address), port 9000. A TCP data channel is established, sending 10 frames/s with 1460 bytes/frame from the active to the passive side. The second parameter block specifies 0 frames/s with 0 bytes/frame (that is, no data) in the opposite direction. The measurement ends when stopped by SIGINT (i.e. Ctrl+C).
One Saturated TCP Flow
Making the TCP flow saturated (that is, to send as much as possible) is as easy as setting the frame rate to 0 frames/s (meaning to send as much as possible). Each frame again has a size of 1460 bytes:
netperfmeter <passive side>:9000 -tcp const0:const1460:const0:const0
A Bidirectional TCP Flow
In order to also transmit data in the direction from passive side to active side, just update the parameters:
netperfmeter <passive side>:9000 -tcp const0:const1460:const10:const1460
A fixed frame size of 1460 bytes may not be useful in all scenarios. Therefore, NetPerfMeter provides random distributions as well:
- exp<average>: Negative exponential distribution with given average. Example: exp1000.
- uniform<lower>,<upper>: Uniform distribution from lower to upper bound. Example: uniform500,25000.
Random distributions can of course be used for the frame rate as well.
NetPerfMeter is also able to turn MPTCP on or off per socket (given that the implementation supports the TCP_MULTIPATH_ENABLE socket option). Then, MPTCP can be enabled by the “cmt=mptcp” option:
netperfmeter <passive side>:9000 -tcp const0:const1460:const0:const0:cmt=mptcp
MPTCP versus TCP
Of course, NetPerfMeter can use multiple data channels as well:
netperfmeter <passive side>:9000 \ -tcp const0:const1460:const0:const0:cmt=mptcp \ -tcp const0:const1460:const0:const0:cmt=off
In this case, NetPerfMeter starts one MPTCP flow and one concurrent TCP flow. “cmt=off” turns MPTCP off (the default; just not specifying the “cmt” option has the same effect).
One of the most important features for researchers is of course to easily get machine-readable results files. Two parameters provide the generation of such files:
- : Generates files named <prefix>-<active|passive>-<flow ID>-<stream ID>.<suffix> with aggregates over the whole measurement. The scalar file format is compatible to OMNeT++ scalar files.
- -vector=<prefix>.<suffix> :
Generates files named <prefix>-<active|passive>-<flow ID>-<stream ID>.<suffix> with per-frame statistics. The vector files are text tables that could for example be processed by GNU R2 or GNU Plot3. For convenience, if the suffix ends with .bz>2, the results file is BZip2-compressed on the fly. To automate measurements, the -runtime=<seconds> option specifies a duration of the measurement. That is, the following example would run a 60s MPTCP versus TCP comparison, and record scalars as well as vectors:
netperfmeter <passive side>:9000 \ -runtime=60 \ -scalar=scalars.sca.bz2 \ -vector=vectors.vec.bz2 \ -tcp const0:const1460:const0:const0:cmt=mptcp \ -tcp const0:const1460:const0:const0:cmt=off
NetPerfMeter is a convenient and flexible open source tool for transport protocol performance analysis. It particularly provides multi-platform support and works with TCP, SCTP, UDP as well as DCCP.
 Dreibholz, T.; Zhou, X. and Fa, F.: “Multi-Path TCP in Real-World Setups – An Evaluation in the NorNet Core Testbed”, in 5th International Workshop on Protocols and Applications with Multi-Homing Support (PAMS), pp. 617–622, Gwangju/South Korea, March 2015.
 Dreibholz, T.; Adhari, H.; Becke, M. and Rathgeb, E. P.: “Simulation and Experimental Evaluation of Multipath Congestion Control Strategies”, in Proceedings of the 2nd International Workshop on Protocols and Applications with Multi-Homing Support (PAMS), Fukuoka/Japan, March 2012.
 Dreibholz, T.: “Evaluation and Optimisation of Multi-Path Transport using the Stream Control Transmission Protocol”, Habilitation Treatise, University of Duisburg-Essen, Faculty of Economics, Institute for Computer Science and Business Information Systems, March 2012.
 Dreibholz, T.; Adhari, H.; Becke, M. and Rathgeb, E. P.: “NetPerfMeter – A Versatile Tool for Multi-Protocol Network Performance Evaluations”, OMNeT++ Code Contribution, University of Duisburg-Essen, Institute for Experimental Mathematics, February 2012.
 Dreibholz, T.; Becke, M.; Adhari, H. and Rathgeb, E. P.: “Evaluation of A New Multipath Congestion Control Scheme using the NetPerfMeter Tool-Chain”, inProceedings of the 19th IEEE International Conference on Software, Telecommunications and Computer Networks (SoftCOM), pp. 1–6, Hvar/Croatia, September 2011.
 Adhari, H.; Dreibholz, T.; Becke, M.; Rathgeb, E. P. and Tüxen, M.: “Evaluation of Concurrent Multipath Transfer over Dissimilar Paths”, in Proceedings of the 1st International Workshop on Protocols and Applications with Multi-Homing Support (PAMS), pp. 708–714, Singapore, March 2011.
In September 2013, Apple surprised the networking community by enabling Multipath TCP on all iOS devices . The main motivation for this deployment was to support Apple’s voice recognition application and enable it to work seamlessly over both WiFi and cellular networks. Multipath TCP is a good match for this application, but it can also be used for other use cases.
At IETF‘93 in Prague, SungHoon Seo provided several very interesting details of the Gigapath commercial service that is now sold by KT. This service enables smartphone users to reach bandwidth of up to 1 Gbps on existing smartphones. This is probably the fastest commercially deployed mobile network. They achieve this high bandwidth by combining both fast LTE (with carrier aggregation) and fast WiFi networks on Multipath TCP enabled smartphones. At this stage, only the Samsung Galaxy S6 and Galaxy S6 Edge smartphones support the Gigapath service, but KT is working with other vendors to add Multipath TCP on their smartphones. Measurements presented at the MPTCP Working Group meeting revealed that current smartphones are able to reach throughputs of about 800 Mbps out of a theoretical maximum of 1.17 Gbps.
What is more impressive is how the system has been implemented and how the users can benefit from it. The figure below, extracted from SungHoon Seo’s presentation provides the general architecture of the GIGA Path system.
On the client side, the smartphones include the open-source Multipath TCP implementation in the Linux kernel. Samsung reused release 0.89.4 and backported it in their Android kernel. The full source code of their Multipath TCP kernel is available online
Enabling Multipath TCP on the smartphone is the first step in deploying it. However, this is not sufficient since there are very few servers that support Multipath TCP today. To enable their users to benefit from Multipath TCP for all the applications that they use, KT has opted for a SOCKSv5 proxy. This proxy is running on x86 servers using release 0.89.5 of the open-source Multipath TCP implementation in the Linux kernel. During the presentation, SungHoon Seo mentioned that despite the recent rollout of the service, there were already 5,500 active users on the SOCKS proxy the last time he checked. Thanks to this proxy, the subscribes of the Giga Path service in Korea can benefit from Multipath TCP with all the TCP-based applications that they use.
At the end of KT’s presentation, another network engineer mentioned that he would come back to his management and propose a similar approach to deploy Multipath TCP in his network. We can thus expect other large scale deployments in the coming months.
Some time ago, @ben_pfaff sent a tweet indicating that he was impressed by a google scholar search that returns more than 1,000 hits for “open vswitch”. This triggered my curiosity and I wondered what was the current impact of Multipath TCP in the scientific literature.
Google scholar clearly indicates that there is a Multipath TCP effect in the academic community. For the “Multipath TCP” query, google scholar already lists more than 1500 results.
For the “mptcp” query, google scholar reports a similar number of documents.
This is a clear indication that there is a growing number of researchers who are adopting Multipath TCP and propose extensions and improvements to it. The researchers who start to work on Multipath TCP will need to understand an already large set of articles. As a first step towards a detailed bibliography on Multipath TCP, I’ve started to collect some notes on IETF documents and scientific paper on this topic. This is a work in progress that will be updated every time I find some time to dig into older papers or new interesting papers are published.
The current version of the annotated bibliography will always be available from https://github.com/obonaventure/mptcp-bib This repository contains all the latex and bibtex files and could be useful to anyone doing research on Multipath TCP. The pdf version (mptcp-bib.pdf) will also be updated after each major change for those who prefer pdf documents.
Various tutorials and trainings have been taught on Multipath TCP during the last years. Some of these have been recorded and are available via youtube.com.
The most recent video presentation is the talk given by Octavian Purdila from intel at the netdev‘01 conference In this talk, Octavian first starts with a brief tutorial on Multipath TCP targeted at Linux networking kernel developpers and then describes in details the structure of the current code and the plans for upstreaming it to the official Linux kernel.
A longer tutorial on the Multipath TCP protocol was given by Olivier Bonaventure at IETF‘87 in Berlinin August 2013.
Christoph Paasch gave a shorter Multipath TCP tutorial earlier during FOSDEM‘13 in Brussels.
Costin : google tech talk