Homage to Icarus

I’m set­ting up an iso­lated net­work for peo­ple to test inter­nal appli­ca­tions on, since the devel­op­ers all have Sun work­sta­tions with a dual-port Gigabit NIC on the moth­er­board, and we’ve got a bunch of older net­work equip­ment that we haven’t got­ten around to eBay­ing yet. What I’m doing is link­ing the sec­ond NICs together with some vir­tual machines and the older net­work equip­ment to cre­ate a sep­a­rate devel­op­ment network.

The devel­op­ment net­work is a full Layer-3 net­work run­ning an IGP between mul­ti­ple nodes with attached client boxes. This allows me to play around with a decent lab net­work, and pro­vides devel­op­ers with a way to dis­cover that Linux sets the TTL of mul­ti­cast pack­ets to “1” well before they are called to explain why their appli­ca­tion didn’t work even after loads of test­ing, spend 8 hours play­ing head-desk, and finally start ques­tion­ing me about fire­walls on our inter­nal net­work, forc­ing me to claw it out of them that they are dri­ving mul­ti­cast with­out a license and explain how to use tcpdump.

Not that I’ve had to do that a dozen times now, or any­thing…

This means I have to con­fig­ure sta­tic routes on the devel­oper work­sta­tions so they can access things in the lab out­side their local sub­net. You start off by con­fig­ur­ing sta­tic routes in your distro’s cho­sen for­mat (this is RHEL5 at work, so it’s /etc/sysconfig/network-scripts/route-ethX), and then you step it up a notch by writ­ing scripts to dis­trib­ute these files, then start using rgang or func, and start think­ing about using your sys­tems pro­gram­ming tool to dis­trib­ute the routes. And then you smack your fore­head and fig­ure out that this is all stu­pid: there is already an IETF stan­dard way to dis­trib­ute net­work con­fig­u­ra­tion which you should be using: DHCP.

There’s even DHCP option 121, which pro­vides a way to dis­trib­ute CIDR infor­ma­tion (mod­ern sta­tic routes) to clients. Unfortunately this stan­dard option isn’t sup­ported out of the box on mod­ern dhclient or ISC dhcpd, so you need to con­fig­ure it and script it in.

First, on the client, /etc/dhclient-exit-hooks

#!/bin/bash
#
# /etc/dhclient-exit-hooks
#
# This file is called from /sbin/dhclient-script after a DHCP run.
#

#
# parse_option_121:
# @argv: the array contents of DHCP option 121, separated by spaces.
# @returns: a colon-separated list of arguments to pass to /sbin/ip route
#
function parse_option_121() {
        result=""

        while [ $# -ne 0 ]; do
                mask=$1
                shift

                # Is the destination a multicast group?
                if [ $1 -ge 224 -a $1 -lt 240 ]; then
                        multicast=1
                else
                        multicast=0
                fi

                # Parse the arguments into a CIDR net/mask string
                if [ $mask -gt 24 ]; then
                        destination="$1.$2.$3.$4/$mask"
                        shift; shift; shift; shift
                elif [ $mask -gt 16 ]; then
                        destination="$1.$2.$3.0/$mask"
                        shift; shift; shift
                elif [ $mask -gt 8 ]; then
                        destination="$1.$2.0.0/$mask"
                        shift; shift
                else
                        destination="$1.0.0.0/$mask"
                        shift
                fi

                # Read the gateway
                gateway="$1.$2.$3.$4"
                shift; shift; shift; shift

                # Multicast routing on Linux
                #  - If you set a next-hop address for a multicast group, this breaks with Cisco switches
                #  - If you simply leave it link-local and attach it to an interface, it works fine.
                if [ $multicast -eq 1 ]; then
                        temp_result="$destination dev $interface"
                else
                        temp_result="$destination via $gateway dev $interface"
                fi

                if [ -n "$result" ]; then
                        result="$result:$temp_result"
                else
                        result="$temp_result"
                fi
        done

        echo "$result"
}

function modify_routes() {
        action=$1
        route_list="$2"

        IFS=:
        for route in $route_list; do
                unset IFS
                /sbin/ip route $action $route
                IFS=:
        done
        unset IFS
}

if [ "$reason" = "BOUND" -o "$reason" = "REBOOT" -o "$reason" = "REBIND" -o "$reason" = "RENEW" ]; then
        # Delete old routes, if they exist
        if [ -n "$old_classless_routes" ]; then
                modify_routes delete "$(parse_option_121 $old_classless_routes)"
        fi

        # Add new routes, if they exist...
        if [ -n "$new_classless_routes" ]; then
                modify_routes add "$(parse_option_121 $new_classless_routes)"
        fi
fi

We use /etc/dhclient-exit-hooks because the RHEL5 dhclient-script only calls the up-hooks script on BOUND and REBOOT, so if you change your sta­tic routes on the server, your client won’t pick them up until the box reboots or the inter­face is oth­er­wise cycled.

The obvi­ous prob­lem here is that it’s always delet­ing the old routes and adding the new routes in two stages, a worth­while enhance­ment for this script is to diff the old and new routes and deter­mine which ones actu­ally need to be removed/added.

So that will not do any­thing at first, because dhclient doesn’t actu­ally read option 121 until you tell it to. For that, you need to edit /etc/dhclient.conf, and tell it how to han­dle option 121 in a way that the script above can understand:

#
# dhclient.conf
#

option classless-routes code 121 = array of unsigned integer 8;
request;

This tells dhclient to read all options, parse option 121 into an array of numeric bytes, and pro­vide that array as a space-separated string as the new_classless_routes and old_classless_routes variables.

So now we’ve got­ten all that taken care of, we need to start dis­trib­ut­ing routes from the DHCP server. For that, you need to update your /etc/dhcpd.conf file:

#
# dhcpd.conf
#

option classless-routes code 121 = array of unsigned integer 8;

subnet 10.23.1.0 netmask 255.255.255.0 {
        [...]
        # Routes for 10.23.0.0/16 via 10.23.1.1, and 224.0.0.0/4 (all IP multicast) via same
        option classless-routes 16,10,23,10,23,1,1,4,224,10,23,1,1
        [...]
}

You can also put that option into a host stanza if you’re doing that. Finally, as I’m using cob­bler, I wanted to be able to have the new “static-routes” inter­face option end up in my cobbler-managed DHCPd con­fig­u­ra­tion. Here’s a bit of my tem­plate that puts that con­fig­u­ra­tion option into the appro­pri­ate DHCP option:

#
# /etc/cobbler/dhcp.template
#

[...]

#for dhcp_tag in $dhcp_tags.keys()
group {
        #for mac in $dhcp_tags[$dhcp_tag].keys():
                #set iface = $dhcp_tags[$dhcp_tag][$mac]
                #if $iface.dns_name
        host $iface.dns_name {
                hardware ethernet $mac;
                        #if $iface.ip_address
                fixed-address $iface.dns_name;
                        #else
                ddns-hostname "${iface.dns_name.split('.')[0]}";
                        #end if
                        #if $iface.static_routes:
                                #set val121=""
                                #for routespec in $iface.static_routes:
                                        #set gateway=$routespec.split(':')[1]
                                        #set destcidr=$routespec.split(':')[0]
                                        #set destnet=$destcidr.split('/')[0]
                                        #set destmask=$destcidr.split('/')[1]
                                        #
                                        #if val121
                                                #set val121=$val121 + ",$destmask"
                                        #else
                                                #set val121=$destmask
                                        #end if
                                        #
                                        #if int($destmask) > 24
                                                #set val121=$val121 + "," + $destnet.replace('.', ',')
                                        #else if int($destmask) > 16
                                                #set val121=$val121 + "," + $destnet.split('.')[0] + "," + $destnet.split('.')[1] + "," + $destnet.split('.')[2]
                                        #else if int($destmask) > 8
                                                #set val121=$val121 + "," + $destnet.split('.')[0] + "," + $destnet.split('.')[1]
                                        #else
                                                #set val121=$val121 + "," + $destnet.split('.')[0]
                                        #end if
                                        #
                                        #set val121=$val121 + "," + $gateway.replace('.', ',')
                                #end for

                option classless-routes $val121
                        #end if
        }
                #end if
        #end for
}

Obviously, there are likely bugs in this script, and I’m only using it on a cou­ple of boxes in my lab net­work, so feel free to point out any issues in the com­ments and I’ll update the above accordingly.