Re: Connection of a qemu guest to the 'net.

[Berto Furth]

Hi Peter,

This is a long email. In answering your question I got on a roll and ended up 
writing an essay. 

In summary, to answer your last question which was "br0 needs a configuration? 
tap0 needs to relate to br0? How?"   tap0 needs to be associated to br0 so that 
your QEMU guest can send traffic over it and the traffic can be processed by 
the host. In addition you can associate other interfaces to br0, like your 
hosts's real/physical eth0 interface, or other tap interfaces connected to 
other guests, so that your guest can communicate directly with the real world 
and/or other guests.

Anyway your email got me thinking and here's what I wrote down. This isn't 
necessarily all directed to you because I suspect you've got a fair bit of 
Linux and networking experience so some of this stuff will already be known to 
you. Let me know if it makes sense or if it's rubbish....



A bridge interface (e.g. br0) is a way for Ethernet traffic to pass unmodified 
and unprocessed between interfaces that are members of the bridge. If you had 
two real Ethernet interfaces on your host system (eth0, eth1), you could make 
your host act like an Ethernet "switch" by configuring your eth0 and eth1 
interfaces to be members of the same bridge interface (br0). Traffic heard on 
eth0 would be forwarded to eth1 and vice versa.  (N.B. Look into bridging or 
switching tables for more details about how forwarding decisions are made). 
This means that the physical Ethernet networks connecting to eth0 and eth1 are 
*different* Ethernet segments, but they will be in the *same* IP subnet.

In this kind of bridging setup eth0 and eth1 do NOT have IP addresses 
configured but the bridge interface (br0) does. This is because the ethX 
interfaces are just acting as conduits. The virtual br0 interface is the means 
that the host will use to perform IP communication to the devices connected to 
both the eth0 and eth1 networks.

When people hear that eth0 and eth1 don't need IP addresses they sometimes get 
a bit confused. Think of a managed Ethernet switch (I'm sure you have one in 
your network if you're the type of person who uses QEMU). You don't give every 
port on your Ethernet switch a separate IP address, right? Instead you give the 
switch a single management IP address that can be accessed via all the ports of 
the switch. In the same way if you wanted to make your host perform network 
bridging between eth0 and eth1, then you wouldn't give both eth0 and eth1 an IP 
address. Instead you give an IP address to the bridge interface (br0) which has 
both eth0 and eth1 as member interfaces. The host uses this single IP address 
configured on the br0 interface to communicate with devices on both Ethernet 
segments.


In QEMU a tap interface (e.g. tap0) is just a conduit that conveys network 
traffic between your guest and the host. It doesn't need an IP address because 
all it's doing is passing traffic. Of course, your QEMU guest still needs to be 
configured with an IP address within the guest operating system itself.

What we need to do is make the tap interface a member of a host bridge 
interface so that the traffic can either be heard by the host, or forwarded off 
to a real Ethernet network, or forwarded off to other guests using other tap 
interfaces (e.g. tap1). We do this by simply making the tap interface a member 
of the bridge interface, and then the guest can communicate with all the 
devices which are connected to the other member interfaces associated with the 
bridge.


I'm going to illustrate by looking at three separate styles of configuration 
using QEMU and tap interfaces. QEMU has the ability to work with preexisting 
tap interfaces, or to dynamically create them at startup. It also has the 
ability to dynamically create bridge interfaces or work with preexiting 
bridges. 

I'm using a setup where QEMU automatically creates tap interfaces when it 
starts and bridge interfaces are preexisting. In my view is the best way to go 
because it means that you can have multiple instances of QEMU running and 
communicating with each other and they can all dynamically generate and destroy 
their own tap interfaces as they need them rather than relying on there being 
pre-configured tap interfaces in place for them all. By keeping the bridge 
configuration separate to QEMU, you are more easily able to customize how the 
bridge interfaces behave.

Note 1 : In the scenarios below I'm only showing the commands to temporarily 
modify host networking behavior. Unless otherwise noted, these changes will not 
survive a host reboot. I'm taking this avenue because the steps for modifying 
network configurations to survive reboots is very specific to each 
distribution. Hopefully if you are able to understand what we're doing with 
these command line examples you can look up how to perform the equivalent 
operations using the configuration tools and files of your specific 
distribution.

Note 2 : In all the command line examples below I first present the modern 
iproute2 (ip) commands, then following that I use the older ifconfig / brctl / 
route commands. You only need to perform one of the two sets of commands to get 
the same result. 

Note 3 : If you get errors running the commands related to creating bridge 
interfaces then you may need to recompile your kernel to include bridging 
support.


Scenario 1 : You only want the guest(s) to be able to communicate with the host 
but not with the real network (simplest)

First we need to create a bridge interface (e.g. br9) on the host and give it 
an IP and IPv6 address that don't conflict with any existing IP subnets 
configured on the host. In this example we'll give the bridge interface an IP 
address of 10.0.9.1 / 255.255.255.0. We also give the interface an IPv6 address.

Note that we can call the bridge anything we like (br0, br54, bridgeA, 
myqemubridge) but brX is the normal interface name format.

root@lfs:~# ip link add br9 type bridge
root@lfs:~# ip link set br9 up
root@lfs:~# ip address add 10.0.9.1/24 dev br9
root@lfs:~# ip address add fd99:9999:9999:9999::1 dev br9

OR

root@lfs:~# brctl addbr br9
root@lfs:~# ifconfig br9 10.0.9.1/24
root@lfs:~# ifconfig br9 inet6 add fdaa:aaaa:aaaa:aaaa::1/64

Next we need to make sure we have a script that will help QEMU automatically 
attach a newly created tap interface to the bridge when QEMU comes up. The 
default name for this script is "qemu-ifup" and there are plenty of great 
examples of what you can put in that script. For example

https://en.wikibooks.org/wiki/QEMU/Networking#qemu-ifup

Below are my very simple versions of the "qemu-ifup" script. The first version 
uses "iproute2", and the second uses the older "ifconfig / brctl" commands. You 
only need to pick one of those. You should also create a "downscript" called 
"qemu-ifdown" that gets executed when QEMU exits and deletes the tap interface, 
but it can just be blank if you're following this guide.


#! /bin/sh
# qemu-ifup script to bring a network (tap) device for qemu up 
# iproute2 version

# You must specify the preexisting bridge interface you want the tap
# connected to in the variable "bridge" set below.
bridge=br9

ip=$(which ip)
echo $0 connecting $1 to $bridge
ip link set "$1" up
ip link set "$1" master "$bridge"
exit

- - - - - - - - - - - - - - - - - - - -

#! /bin/sh
# qemu-ifup script to bring a network (tap) device for qemu up 
# ifconfig / brctl version

# You must specify the preexisting bridge interface you want the tap
# connected to in the variable "bridge" set below.
bridge=br9

brctl=$(which brctl)
echo $0 connecting $1 to $bridge
ifconfig "$1" 0.0.0.0 up
brctl addif $bridge "$1"
exit

- - - - - - - - - - - - - - - - - - - -

#! /bin/sh
# qemu-ifdown script to execute just before the tap interface is removed.
# Normally this script is empty.
echo $0 removing tap interface $1
exit


Remember to make the scripts executable with a command like

chmod a+x qemu-ifup qemu-ifdown


Finally run QEMU with a command line that enables tap networking. For example

qemu -machine [whatever] ...... -nic tap

qemu -machine [whatever] ...... -netdev 
type=tap,id=testnet,script="./qemu-ifup",downscript="./qemu-ifdown" -global 
driver=[whatever-ethernet-driver-you-use],property=netdev,value="testnet" 


After the guest is running you should make sure your guest operating system is 
configured with an IP address in the same subnet as the br9 interface on the 
host. In this case an example of a valid guest IP address would be 
10.0.9.101/24. The guest would also typically be configured to use the host's 
br9 address (10.0.9.1) as a default gateway. After this the guest and host 
should be able to have full network communication with each other, but the 
guest won't be able to connect with hosts in the "real" network. The same 
should go for other guests you launch in other QEMU sessions on the same host.


Scenario 2 : You want the guest(s) to get direct (bridged) access to the host's 
real Ethernet network. (Most useful in my view)

This is similar to scenario 1 except you must also make your real host Ethernet 
interface (e.g. eth0) a member of the same bridge as the tap interfaces.

This means that you need to reconfigure your system so that eth0 no longer has 
an IP address. This is because the eth0 interface will now act as a mere 
conduit just like the tap interfaces. Your nominated bridge interface will get 
the IP address associated with the host on the real Ethernet network. 

In most cases real host Ethernet interfaces get addresses via DHCP however 
since that's configured and done in a wide variety of ways depending on your 
distro I'm going to assume here that your eth0 is currently using a static IP 
address. That being said it should not be hard for you to find out how to 
configure your bridge interface as a DHCP client in your specific distro if 
that's what you would like.

First you need to remove the IP address from eth0. Next create a new bridge 
interface (e.g. br7), then associate your eth0 interface with it. Finally you 
need to move the IP configuration that was on eth0 over to the new bridge 
interface (br7). In the example below eth0 originally has an ip address of 
10.0.1.22/24 so we move that to the br7 interface. The IPv6 addresses will take 
care of themselves via IPv6 auto configuration (SLAAC) assuming that's enabled 
in your real network. The other thing we do in this example is reconfigure the 
host's default gateway that was in place (10.0.1.1) as this will be 
automatically removed when we delete eth0's IP address.

Also note that at the point where you remove eth0's IP address the host will 
lose network connectivity via eth0. For that reason it's probably best to 
perform these commands on the host's console or via a different network 
interface.

root@lfs:~# ip address show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group 
default qlen 1000
    link/ether dc:XX:XX:XX:XX:55 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.22/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
     inet6 2001:XXXX:XXXX:XXXX:deXX:XXff:feXX:XX55/64 scope global dynamic 
mngtmpaddr
       valid_lft 2591816sec preferred_lft 604616sec
    inet6 fe80::deXX:XXff:feXX:XX55/64 scope link
       valid_lft forever preferred_lft forever
root@lfs:~# ip route
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.22
root@lfs:~# ip address del 10.0.1.22/24 dev eth0
root@lfs:~# ip link add br7 type bridge
root@lfs:~# ip link set br7 up
root@lfs:~# ip link set eth0 master br7
root@lfs:~# ip address add 10.0.1.22/24 dev br7
root@lfs:~# ip route add default via 10.0.1.1

OR

root@lfs:~# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500  metric 1
        inet 10.0.1.22  netmask 255.255.255.0  broadcast 10.0.1.255
        inet6 2001:XXXX:XXXX:XXXX:deXX:XXff:feXX:XX55  prefixlen 64  scopeid 
0x0<global>
        inet6 fe80::deXX:XXff:feXX:XX55  prefixlen 64  scopeid 0x20<link>
        ether dc:XX:XX:XX:XX:55  txqueuelen 1000  (Ethernet)
        RX packets 847  bytes 71856 (70.1 KiB)
        RX errors 0  dropped 6  overruns 0  frame 0
        TX packets 137  bytes 17757 (17.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@lfs:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.1.1        0.0.0.0         UG    0      0        0 eth0
10.0.1.0        *               255.255.255.0   U     0      0        0 eth0

root@lfs:~# ifconfig eth0 0.0.0.0
root@lfs:~# brctl addbr br7
root@lfs:~# brctl addif br7 eth0
root@lfs:~# ifconfig br7 10.0.1.22/24
root@lfs:~# route add default gw 10.0.1.1


After running these commands your host should have regained connectivity to the 
real network however eth0 will just be a "conduit" interface and all the IP 
configuration will now belong to bridge interface br7. 

We can now connect QEMU guest tap interfaces to the bridge br7 in the same way 
as in Scenario 1. All we need to do is modify the "qemu-ifup" script to use 
"bridge=br7" instead of "br9" which we used in Scenario 1. Guests should now 
have full access to the "real" Ethernet network that eth0 is physically 
connected to.

If you like, you can keep multiple kinds of bridge networks (e.g. "br9" and 
"br7") active on your host so that you can launch guests connecting to either 
type of bridged network depending on your requirements.


Scenario 3 : You want the host to act as a router between a "guest subnet" and 
the "real" network. (complicated)

In this scenario guests will be able to access the "real" network but they will 
be on a different IP subnet to the eth0 network. The host will have 
responsibility for routing / forwarding IP packets between the "guest subnet" 
and the rest of the world. This also means that a router on the real network 
will need to be reconfigured so that it knows to route packets to the guest 
subnet via the host's eth0 interface. If the router on the real network is not 
reconfigured to know how to route to the guest network then packets will make 
it from QEMU guests to real devices, but then packets transmitted from devices 
in the real network won't make it back to the guests.

In this scenario we first need to make sure that the host is configured for IP 
routing / IP forwarding. 

This can be done on the command line as follows for both IPv4 and IPv6

root@lfs:~#echo 1 > /proc/sys/net/ipv4/ip_forward
root@lfs:~#echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

On many (but not all) systems you can make this change permanent across reboots 
by editing /etc/sysctl.conf to include the lines

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding=1

Next, we need to proceed exactly as in Scenario 1 so I won't rehash those steps 
here.

Let's say we have created bridge interface br9 with IP address 10.0.9.1/24 as 
per the example in Scenario 1. Let's also say that our eth0 interface has an IP 
address of 10.0.1.22.

This means that on our external router on the "real" network we need to 
configure a route for network 10.0.9.1/24 via 10.0.1.22. In cisco terminology 
that's

ip route 10.0.1.0 255.255.255.0 10.0.9.1

You'd also need to do the same for the IPv6 network if you're using IPv6.


Note that you can extend Scenario 3 (and Scenario 1) by making the host a DHCP 
server for the guests. In addition the "radvd" daemon can be installed on the 
host so that guests can be automatically configured for IPv6. Setting these 
services up is not normally hard, and makes things very convenient in regards 
to not having to manually configure the IP stacks on your guests. However since 
configuring and installing these tools is quite specific to each distro we 
won't discuss it here. The only thing I want to stress about DHCP and radvd is 
that you only set them up to work on your bridge interface which connect to 
guests and NOT on your Ethernet interfaces that connect to the "real" network. 
If you accidentally get these services running on an interface connecting to 
your real network they will conflict with the already running services and your 
real network will be in chaos!!

There's no need for setting up your host as a DHCP or "radvd" server in 
Scenario 2 because your guests will have access to these services via the real 
network where these services should already be working and active.




I hope some of this helps Peter. All the best.

Berto.


On Wed, 17 Mar 2021, at 03:30, peter@easthope.ca wrote:
> Hello Berto,
> 
> Thanks for the reply.
> 
> In the following, joule is the qemu host.
> 
> From: Berto Furth
> Date:         Tue, 16 Mar 2021 21:18:08 +1100
> > I'm assuming you're also using dynamically created "tap" interfaces in your 
> > setup. 
> 
> tap0 is created in the qemu host when it boots, by a line in 
> /etc/crontab. I don't do anything to remove tap0.  Therefore tap0 
> will exist until something catastrophic happens or the system shuts down.
> 
> peter@joule:/home/peter$ grep tap0 /etc/crontab
> @reboot root ip tuntap add mode tap tap0
> 
> > That is, when QEMU starts it's creating a tap interface to funnel 
> > Ethernet traffic to and from the guest...so a command line something like
> >
> > -netdev 
> > type=tap,id=testnet,script="./qemu-tap-up",downscript="./qemu-tap-down"
> 
> There is no file "./qemu-tap-up" or similar.  There is no script 
> option in the qemu command.  /etc/network/interfaces has a stanza for 
> tap0.  tap0 exists and has an ip address.
> 
> peter@joule:/home/peter$ ip addr show tap0
> 6: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP 
> gr
> oup default qlen 1000
>     link/ether 56:06:3e:f1:4e:13 brd ff:ff:ff:ff:ff:ff
>     inet 172.23.6.1/24 brd 172.23.6.255 scope global tap0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::5406:3eff:fef1:4e13/64 scope link
>        valid_lft forever preferred_lft forever
> 
> > You can do routing, but you would still need to use a bridge interface for 
> > the 
> > dynamically created "tap" interface to connect to!! 
> 
> Accepted.  Thanks.
> 
> Incidental note.
> I have a LAN with a Linux router machine and a machine on a subnet.  
> The router has Shorewall and dnsmasq.  Routing from the subnetted 
> machine to the router host and to the Internet works.  No bridge is 
> required for that.
> 
> > It's just that rather than setting up the bridge interface to do 
> > bridging between the tap and the "real" ethernet of the host, you'd 
> > set up the bridge interface to be a separate routed interface on your 
> > host.
> 
> I don't really understand that distinction.  According to 
> https://en.wikipedia.org/wiki/Network_bridge "A network bridge is a 
> computer networking device that creates a single, aggregate network 
> from multiple communication networks or network segments. This 
> function is called network bridging."
> 
> Apparently qemu requires a bridge to connect the guest.  Communication 
> can not be routed between guest and host without a bridge.  (?)
> 
> br0 is created in the qemu host by another line in /etc/crontab.
> 
> peter@joule:/home/peter$ grep br0 /etc/crontab
> @reboot root ip link add name br0 type bridge
> 
> br0 exists analogous to tap0 but does not have an IP address.
> 
> peter@joule:/home/peter$ ip addr show br0
> 7: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group 
> default qlen 1000
>     link/ether 76:da:52:dd:52:45 brd ff:ff:ff:ff:ff:ff
> 
> > Routing is a bit more difficult to setup than simple bridging because 
> > you have to first make sure your host is configured for routing (most 
> > aren't by default) and then you have to make sure your "real" 
> > networking infrastructure knows about the new QEMU guest subnet and 
> > how to route to it (via your host).
> > 
> > See the following URL for an example script to run when your QEMU tap 
> > interface is created that allows routing and you'll see how much extra 
> > effort it is to get routing (in this case with NAT) working compared 
> > to simple bridging.
> > 
> > https://wiki.qemu.org/Documentation/Networking/NAT
> 
> This linux router has shorewall which has a configuration file 
> /etc/shorewall/snat containing this line.
> MASQUERADE  172.23.6.0/24  $NET_IF
> 
> Other subnets work that way.
> 
> If tap0 is configured with address 172.23.6.2 in 
> /etc/network/interfaces, SNAT should work for the qemu guest.
> 
> > If you just want QEMU to do simple NAT between the guest and the 
> > "real" network then just use "SLIRP" as per
> > 
> > https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
> 
> Will use SLIRP if necessary.  Not sure it's necessary.
> 
> > (I haven't tested it...I always use tap and bridging)
> 
> I want to do that also.  From the above you see that tap0 and br0 
> exists in the qemu host here. dnsmasq provides DNS.  Shorewall 
> provides SNAT.  The qemu command has this option.
> 
> -nic tap,model=ne2k_pci
> 
> Nevertheless no connection.
> 
> peter@joule:/home/peter$ ping 172.23.6.1
> PING 172.23.6.1 (172.23.6.1) 56(84) bytes of data.
> 64 bytes from 172.23.6.1: icmp_seq=1 ttl=64 time=0.043 ms
> 
> --- 172.23.6.1 ping statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms
> 
> peter@joule:/home/peter$ ping 172.23.6.2
> PING 172.23.6.2 (172.23.6.2) 56(84) bytes of data.
> From 172.23.6.1 icmp_seq=1 Destination Host Unreachable
> 
> --- 172.23.6.2 ping statistics ---
> 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
> 
> At least one ingredient is missing.  
> 
> br0 needs a configuration?  
> 
> tap0 needs to relate to br0?  How?
> 
> Ideas welcome.
> 
> Thanks,                             ... P.
> 
> -- 
> cell: +1 236 464 1479            Bcc: peter at easthope. ca
> VoIP: +1 604 670 0140
> 
> 
>