Want to create a really interesting virtual network on your own host and test it with ordinary applications? Great! Here is how.

We will make the address 192.168.100.1 act like 127.0.0.1 but route packets through a custom network topology before processing them.

First start with any custom topology. In this example: west and east are endpoints with an Open vSwitch bridge ovs in between. (This would be great for applying OpenFlow rules to packets sent between local applications.)

1
2
3
4
5
6
7
8
9
10

ip link add west type veth peer name west_to_ovs
ip link add east type veth peer name east_to_ovs

ovs-vsctl add-br ovs
ovs-vsctl add-port ovs west_to_ovs
ovs-vsctl add-port ovs east_to_ovs

for i in west west_to_ovs east east_to_ovs ovs; do
  ip link set $i up
done

Now assign addresses and routes for these interfaces. Packets sent to 192.168.100.1 should first be routed into interface west then switched via ovs and finally delivered to east for processing.

1
2
3
4
5

ip addr add 192.168.100.2 dev west
ip addr add 192.168.100.1 dev east

ip route add 192.168.100.1 dev west
ip route add 192.168.100.2 dev east

The ingredients are in place but they don’t work yet. If you ping 192.168.100.1 then the packets are sent to lo instead of being routed through the bridge.

And that brings us to the trick: Policy Routing.

First make Linux globally “forget” that these addresses are local.

1
2

ip route del 192.168.100.2 table local
ip route del 192.168.100.1 table local

Now packets sent to 192.168.100.1 do get routed down the right path. They are not processed at the other end though, because Linux does not remember they are local. We are half way there.

Next create separate routing tables strictly for when packets are received after they have traversed the switch. These tables remember that the addresses are local.

1
2
3
4
5
6
7

ip rule add iif west lookup 100
ip route add local 192.168.100.2 dev west table 100
echo 1 > /proc/sys/net/ipv4/conf/west/accept_local

ip rule add iif east lookup 101
ip route add local 192.168.100.1 dev east table 101
echo 1 > /proc/sys/net/ipv4/conf/east/accept_local

Now we are done!

If you connect to 192.168.100.1 then your packets will first traverse the bridge and then be processed locally. The setup is symmetric so that return traffic will be routed back through the bridge too. This will work with all your favourite programs like ping, curl, apache, etc. Check it out by running tcpdump on west or east.

Go ahead and create interesting virtual networks on your own machine.