SUMMARY
Routing Policies for Multi-NIC Appliances
ISSUE
For a deployment where a VBA (usually the Presentation and/or Management VBA) is configured to use two networks, the network active on interface eth0 will provide the default route for any address without an explicit route. As an example (Figure1):
Figure 1:
The VmWare/Xen/HyperV host is using 10.10.95.0/24.
The Presentation VBA (one NIC, eth0) is using 10.10.133.0/24.
The Management VBA (two NICs); eth0 is using 10.10.133.0/24, eth1 is using 172.28.0.0/24.
Engine VBA One (one NIC, eth0) is using 10.10.133.0/24.
Engine VBA Two (one NIC, eth0) is using 172.28.0.0/24.
The routing table on the Presentation and Engine One VBAs will look like the following (assuming 10.10.133.1 is the gateway):
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.133.1 0.0.0.0 UG 0 0 0 eth0 10.10.133.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
The routing table on the Management VBA will look as follows (using the above assumption for eth0 and 172.28.0.1 as the gateway for eth1):
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.133.1 0.0.0.0 UG 0 0 0 eth0 0.0.0.0 172.28.0.1 0.0.0.0 UG 100 0 0 eth1 10.10.133.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 172.28.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
or
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.10.133.1 0.0.0.0 UG 0 0 0 eth0 10.10.133.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 172.28.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
The routing table for Engine Two VBA will look as follows:
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.28.0.1 0.0.0.0 UG 0 0 0 eth0 172.28.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
Given that all Engine VBAs need network connectivity to the host (10.10.95.0/24 in the above example), the route to the host network:
- must be provided by the default gateway
or - must be explicitly added (i.e. sudo route add -net 10.10.95.0/24 gw 172.28.0.1)
If the latter is chosen, the preferred method of persisting the route through a reboot is to include the 'route add' command in /etc/rc.local, as opposed to using the 'up' or 'post-up' stanza in /etc/network/interfaces (as shown here: http://www.cyberciti.biz/faq/setting-up-an-network-interfaces-file/).