Categories
Mastering Development System & Network

How to debug packet loss on QEMU/KVM virtual machine which uses multiple VLAN interfaces on single network card

I have a server running under Debian 10. There’s one (for now) virtual machine under QEMU/KVM. Under some circumstances, server has only one network card, but I need VM to have access to two different networks: 172.16.0.1/24 (internal) and 93.xx.xx.211/29 (external). I managed to set up two VLAN interfaces using systemd-networkd and bridged them into VM. Here are configuration files for that interfaces:

/etc/systemd/network/enp2s0f0.30.netdev

[NetDev]
Name=enp2s0f0.30
Kind=vlan

[VLAN]
Id=30

/etc/systemd/network/enp2s0f0.30.network

[Match]
Name=enp2s0f0.30

[Network]
DHCP=no
Bridge=bridge0

/etc/systemd/network/enp2s0f0.500.netdev

[NetDev]
Name=enp2s0f0.500
Kind=vlan

[VLAN]
Id=500

/etc/systemd/network/enp2s0f0.500.network

[Match]
Name=enp2s0f0.500

[Network]
DHCP=no
Bridge=bridge1

Here are bridges configuration files:

/etc/systemd/network/bridge

bridge0.netdev   bridge0.network  bridge1.netdev   bridge1.network  

/etc/systemd/network/bridge0.netdev

[NetDev]
Name=bridge0
Kind=bridge

/etc/systemd/network/bridge0.network

[Match]
Name=bridge0

[Network]
Address=172.16.0.100/24    # had to set ip and gateway here
Gateway=172.16.0.1         # because otherwise host wasn't accessible by network
DHCP=no

/etc/systemd/network/bridge0.netdev

[NetDev]
Name=bridge0
Kind=bridge

/etc/systemd/network/bridge1.netdev

[NetDev]
Name=bridge1
Kind=bridge

/etc/systemd/network/bridge1.network

[Match]
Name=bridge1

[Network]
DHCP=no

There’re no any rules configured in host’s iptables, all the policies are ACCEPT.

Here’s network section of VM’s xml:

<interface type='bridge'>
  <mac address='52:54:00:9c:2f:d6'/>
  <source bridge='bridge0'/>
  <target dev='vnet0'/>
  <model type='virtio'/>
  <alias name='net0'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<interface type='bridge'>
  <mac address='52:54:00:f7:c6:13'/>
  <source bridge='bridge1'/>
  <target dev='vnet1'/>
  <model type='virtio'/>
  <alias name='net1'/>
  <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</interface>

And here’s VM’s network configuration:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:9c:2f:d6 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.101/24 brd 172.16.0.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::1012:ff:fec9:f2d6/64 scope link 
       valid_lft forever preferred_lft forever
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:f7:c6:13 brd ff:ff:ff:ff:ff:ff
    inet 93.xx.xx.211/29 brd 93.xx.xx.215 scope global enp7s0
       valid_lft forever preferred_lft forever
    inet6 fe80::1012:ff:fe7f:6c13/64 scope link 
       valid_lft forever preferred_lft forever

ip route sh
default via 93.xx.xx.209 dev enp7s0 onlink 
172.16.0.0/24 via 172.16.0.1 dev enp1s0 
172.16.0.0/24 dev enp1s0 proto kernel scope link src 172.16.0.101 
93.xx.xx.208/29 dev enp7s0 proto kernel scope link src 93.xx.xx.211 

VM’s iptables rules:

iptables -L 
Chain INPUT (policy DROP)
target     prot opt source               destination         
f2b-sshd   tcp  --  anywhere             anywhere             multiport dports ssh
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere             multiport dports http,https

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain f2b-sshd (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere   

The problem is when I ping or trying to access anything in the internal network (172.16.0.1/24) everything works like a charm, not a single packet get lost. But when I’m trying to reach anything on the internet, or even gateway of external network (93.xx.xx.209) I experience heavy packet loss, up to 60%:

....
64 bytes from 8.8.8.8: icmp_seq=38 ttl=111 time=18.1 ms
64 bytes from 8.8.8.8: icmp_seq=39 ttl=111 time=18.0 ms
64 bytes from 8.8.8.8: icmp_seq=40 ttl=111 time=18.0 ms
20/40 packets, 50% loss, min/avg/ewma/max = 17.997/18.067/18.066/18.298 ms
....

I’ve reached up to our ISP, but they stated that everything was fine on their side. Even though there’s a great chance I messed up with some settings, I don’t fully trust them. Is there a way to debug somehow the cause of packet loss. Or maybe I did something wrong, or should I play with MTU/MRU settings of interface or anything else.

I’m completely lost here, hope for your help!
Sorry if I made any mistakes, English is not my first language.

Leave a Reply

Your email address will not be published. Required fields are marked *