Analyzing network performance

Since I am doing more data intensive work lately (Deep Learning, Computer Vision models with hundreds of terabytes of data), I needed to analyze my current network setup both at home and at work and identify the bottlenecks.

Ideally, I wanted to use iperf3 to measure throughput and a more recent tool developed by Apple called networkQuality that uses a more actionable metric about the current state of the network called RPM (Round-trips Per Minute) as this should better measure user experience specifically when the network is under working conditions.

I also wanted to have these running as background services and since I am using Synology for backup both at home and at work and the setup of these won’t change all too often due to being critical infrastructure, they were perfect candidates.

Installing iperf3 on Synology NAS (DSM 7.2+)

In order to install iperf3 on the Synology NAS we have to add another package repository called SynoCommunity. For this we have to log into the DSM, Package Center -> Add the repository there and then install the package SynoCli Monitor Tools from the newly added repository. This also installs the iperf3 commandline tool. We want this to start and run automatically even after rebooting the DSM. For this we can add another systemd service containing the following

$ sudo vi /etc/systemd/system/iperf3.service
[Unit]
Description=Run iperf3 at startup
[Service]
Type=simple
ExecStart=/usr/local/bin/iperf3 -s

[Install]
WantedBy=multi-user.target

$ sudo systemctl start iperf3.service
$ sudo systemctl status iperf3.service
$ sudo systemctl enable iperf3.service

The -s is for running in server mode, listening to new connections. We can now test against this backend from another machine by specifying the endpoint:

$ iperf3 -c wolf-synology.fritz.box
Connecting to host wolf-synology.fritz.box, port 5201
[  7] local 192.168.178.53 port 55961 connected to 192.168.178.23 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec  18.0 MBytes   151 Mbits/sec
[  7]   1.00-2.00   sec  15.3 MBytes   128 Mbits/sec
[  7]   2.00-3.00   sec  20.6 MBytes   172 Mbits/sec
[  7]   3.00-4.00   sec  15.0 MBytes   126 Mbits/sec
[  7]   4.00-5.00   sec  18.6 MBytes   156 Mbits/sec

Installing networkQuality server

NetworkQuality is a tool that comes preinstalled with recent macOS versions and uses a configuration provided by Apple everytime it is called without any additional options

$ networkQuality
==== SUMMARY ====
Uplink capacity: 33.018 Mbps
Downlink capacity: 167.554 Mbps
Responsiveness: Low (454.545 milliseconds | 132 RPM)
Idle Latency: 17.625 milliseconds | 3529 RPM

This is fine for measuring the up and downlinks from the localhost to an endpoint in the world wide web, but for my purposes I needed to measure the performance between to endpoints in the same network (and also respecting parallel connections which most other tools do not bother with).

There exist at least two reference implementations (in Swift and Go). I used the latter which can be found here. Since I wanted this to be running in the background on my NAS, I needed to cross compile the networkqualityd binary using a GCC cross compile toolchain:

$ brew install FiloSottile/musl-cross/musl-cross
$ CC=x86_64-linux-musl-gcc \\
  CXX=x86_64-linux-musl-g++ \\
  GOARCH=amd64 \\
  GOOS=linux \\
  CGO_ENABLED=1 \\
  go build -ldflags "-linkmode external -extldflags -static"

Afterwards, I did copy the binary over to my Synology and created another systemd service as before:

$ sudo vi /etc/systemd/system/networkqualityd.service

[Unit]
Description=Run networkqualityd at startup
[Service]
Type=simple
ExecStart=/usr/local/bin/networkqualityd -create-cert -config-name "wolf-synology.fritz.box" -enable-http2 -listen-addr "0.0.0.0"

[Install]
WantedBy=multi-user.target

With this I now could test against a local endpoint and do some further experiments to optimize the result:

$ networkQuality -C https://wolf-synology.fritz.box:4043/.well-known/nq -k
==== SUMMARY ====
Uplink capacity: 68.958 Mbps
Downlink capacity: 108.231 Mbps
Responsiveness: Low (487.805 milliseconds | 123 RPM)
Idle Latency: 8.167 milliseconds | 7500 RPM

Clearly, these measurements have some room for potential which I will discuss in another upcoming post.

I’m available for software consultancy, training and mentoring. Please contact me, if you are interested in my services.