I’ve been using OpenVZ for a number of years. It’s a good solution for partitioning a server up into several containers, or virtual machines. I also use Xen, but OpenVZ seems easier and slightly more lightweight for some uses.

A while ago I moved to using veth bridged networking. As far as I am concerned, it’s neater than the standard venet proxy-arp type networking, more similar to Xen, and easily allows the network configuration to be able to be set up within the container rather than held in a configuration file outside. It also works well for IPv6, meaning I didn’t have to learn how to get more OpenVZ scripts configured.

However, I’ve had a niggling bug for a long time that would bite every now and then, but didn’t seem to be worthwhile fixing. Upon stopping a container, I’d often end up with no network access to the host machine for a while. Sometimes it was as little as a couple of seconds, other times up to a minute or two. It seemed like the bridge was going back into learning mode, and yet it didn’t seem like it either. There was nothing to indicate this in the kernel log, either. As I don’t stop containers that often, it wasn’t something I bothered to look at.

Today while setting up a new server, I set about trying to work out this issue. A bit of digging led to a tcpdump and watching the packets on the network. It turned out that the server seemed to be unreachable from elsewhere until it responded to an ARP request. That’s odd, though, because the MAC address hasn’t changed on the server, and it’s in other hosts ARP tables before then. However, ARP fits in with the different time delays that are seen.

A bit more staring at the tcpdump output in Wireshark indicated an IP address clash. Hang on a sec, that’s not right. What we then find out is that the server has been talking on the network with the MAC address of the host side interface of the container that was removed, not with its own eth0 MAC address. Obviously that breaks networking when the container’s host interface is ripped out.

But why? Surely the MAC address of the bridge is fixed? A quick check confirms that it is not. It starts with the MAC address of the eth0 interface, then changes to the MAC of the container’s host interface. How odd. Then I discover an enlightening mailing list post that gives light to the issue. It turns out that the bridge takes the lowest MAC address of all its sub-interfaces. This doesn’t matter when a new interface is added, as both MACs are on the machine, but breaks when an interface is remoevd. We have the reason for the problem, now just to find out a good solution.

Why, also, does Xen never seem to have a problem? Looking on a Xen server now easily shows the solution. They use fe:ff:ff:ff:ff:ff for all host-side interfaces, which will never be less than the MAC address already on the bridge. This should be safe, as these will never actually be seen on the wire. However, OpenVZ generates, by default, a MAC address from SWsoft’s own prefix 00:18:51. In the case of this particular machine, the eth0 interface MAC begins with 1c:c1, which gets trumped by the lower SWsoft one.

In the end it’s quite easy to fix – edit the container’s config file in /etc/vz/conf and change the NETIF line so that it ends host_mac=fe:ff:ff:ff:ff:ff. Problem solved, no dropping off the network any more!

It looks like with vzctl versions greater than 3.0.22, an interface can be added with

vzctl set --netif_add eth0,,,fe:ff:ff:ff:ff:ff

However, I’m on 3.0.22, which has no easy way of setting just the host mac without setting the container mac as well, which I’d like to be autogenerated. In this case a quick sed on the config file will resolve the problem.

vzctl set 100 --netif_add eth0 --save

sed -i~ -e 's/^(NETIF.*host_mac=).*$/1fe:ff:ff:ff:ff:ff"/' /etc/vz/conf/100.conf