The development network is a full Layer-3 network running an IGP between multiple nodes with attached client boxes. This allows me to play around with a decent lab network, and provides developers with a way to discover that Linux sets the TTL of multicast packets to “1” well before they are called to explain why their application didn’t work even after loads of testing, spend 8 hours playing head-desk, and finally start questioning me about firewalls on our internal network, forcing me to claw it out of them that they are driving multicast without a license and explain how to use tcpdump.

Not that I’ve had to do that a dozen times now, or anything…

This means I have to configure static routes on the developer workstations so they can access things in the lab outside their local subnet. You start off by configuring static routes in your distro’s chosen format (this is RHEL5 at work, so it’s /etc/sysconfig/network-scripts/route-ethX), and then you step it up a notch by writing scripts to distribute these files, then start using rgang or func, and start thinking about using your systems programming tool to distribute the routes. And then you smack your forehead and figure out that this is all stupid: there is already an IETF standard way to distribute network configuration which you should be using: DHCP.

There’s even DHCP option 121, which provides a way to distribute CIDR information (modern static routes) to clients. Unfortunately this standard option isn’t supported out of the box on modern dhclient or ISC dhcpd, so you need to configure it and script it in.

First, on the client, /etc/dhclient-exit-hooks

#!/bin/bash
#
# /etc/dhclient-exit-hooks
#
# This file is called from /sbin/dhclient-script after a DHCP run.
#

#
# parse_option_121:
# @argv: the array contents of DHCP option 121, separated by spaces.
# @returns: a colon-separated list of arguments to pass to /sbin/ip route
#
function parse_option_121() {
        result=""

        while [ $# -ne 0 ]; do
                mask=$1
                shift

                # Is the destination a multicast group?
                if [ $1 -ge 224 -a $1 -lt 240 ]; then
                        multicast=1
                else
                        multicast=0
                fi

                # Parse the arguments into a CIDR net/mask string
                if [ $mask -gt 24 ]; then
                        destination="$1.$2.$3.$4/$mask"
                        shift; shift; shift; shift
                elif [ $mask -gt 16 ]; then
                        destination="$1.$2.$3.0/$mask"
                        shift; shift; shift
                elif [ $mask -gt 8 ]; then
                        destination="$1.$2.0.0/$mask"
                        shift; shift
                else
                        destination="$1.0.0.0/$mask"
                        shift
                fi

                # Read the gateway
                gateway="$1.$2.$3.$4"
                shift; shift; shift; shift

                # Multicast routing on Linux
                #  - If you set a next-hop address for a multicast group, this breaks with Cisco switches
                #  - If you simply leave it link-local and attach it to an interface, it works fine.
                if [ $multicast -eq 1 ]; then
                        temp_result="$destination dev $interface"
                else
                        temp_result="$destination via $gateway dev $interface"
                fi

                if [ -n "$result" ]; then
                        result="$result:$temp_result"
                else
                        result="$temp_result"
                fi
        done

        echo "$result"
}

function modify_routes() {
        action=$1
        route_list="$2"

        IFS=:
        for route in $route_list; do
                unset IFS
                /sbin/ip route $action $route
                IFS=:
        done
        unset IFS
}

if [ "$reason" = "BOUND" -o "$reason" = "REBOOT" -o "$reason" = "REBIND" -o "$reason" = "RENEW" ]; then
        # Delete old routes, if they exist
        if [ -n "$old_classless_routes" ]; then
                modify_routes delete "$(parse_option_121 $old_classless_routes)"
        fi

        # Add new routes, if they exist...
        if [ -n "$new_classless_routes" ]; then
                modify_routes add "$(parse_option_121 $new_classless_routes)"
        fi
fi

We use /etc/dhclient-exit-hooks because the RHEL5 dhclient-script only calls the up-hooks script on BOUND and REBOOT, so if you change your static routes on the server, your client won’t pick them up until the box reboots or the interface is otherwise cycled.

The obvious problem here is that it’s always deleting the old routes and adding the new routes in two stages, a worthwhile enhancement for this script is to diff the old and new routes and determine which ones actually need to be removed/added.

So that will not do anything at first, because dhclient doesn’t actually read option 121 until you tell it to. For that, you need to edit /etc/dhclient.conf, and tell it how to handle option 121 in a way that the script above can understand:

#
# dhclient.conf
#

option classless-routes code 121 = array of unsigned integer 8;
request;

This tells dhclient to read all options, parse option 121 into an array of numeric bytes, and provide that array as a space-separated string as the new_classless_routes and old_classless_routes variables.

So now we’ve gotten all that taken care of, we need to start distributing routes from the DHCP server. For that, you need to update your /etc/dhcpd.conf file:

#
# dhcpd.conf
#

option classless-routes code 121 = array of unsigned integer 8;

subnet 10.23.1.0 netmask 255.255.255.0 {
        [...]
        # Routes for 10.23.0.0/16 via 10.23.1.1, and 224.0.0.0/4 (all IP multicast) via same
        option classless-routes 16,10,23,10,23,1,1,4,224,10,23,1,1
        [...]
}

You can also put that option into a host stanza if you’re doing that. Finally, as I’m using cobbler, I wanted to be able to have the new “static-routes” interface option end up in my cobbler-managed DHCPd configuration. Here’s a bit of my template that puts that configuration option into the appropriate DHCP option:

#
# /etc/cobbler/dhcp.template
#

[...]

#for dhcp_tag in $dhcp_tags.keys()
group {
        #for mac in $dhcp_tags[$dhcp_tag].keys():
                #set iface = $dhcp_tags[$dhcp_tag][$mac]
                #if $iface.dns_name
        host $iface.dns_name {
                hardware ethernet $mac;
                        #if $iface.ip_address
                fixed-address $iface.dns_name;
                        #else
                ddns-hostname "${iface.dns_name.split('.')[0]}";
                        #end if
                        #if $iface.static_routes:
                                #set val121=""
                                #for routespec in $iface.static_routes:
                                        #set gateway=$routespec.split(':')[1]
                                        #set destcidr=$routespec.split(':')[0]
                                        #set destnet=$destcidr.split('/')[0]
                                        #set destmask=$destcidr.split('/')[1]
                                        #
                                        #if val121
                                                #set val121=$val121 + ",$destmask"
                                        #else
                                                #set val121=$destmask
                                        #end if
                                        #
                                        #if int($destmask) > 24
                                                #set val121=$val121 + "," + $destnet.replace('.', ',')
                                        #else if int($destmask) > 16
                                                #set val121=$val121 + "," + $destnet.split('.')[0] + "," + $destnet.split('.')[1] + "," + $destnet.split('.')[2]
                                        #else if int($destmask) > 8
                                                #set val121=$val121 + "," + $destnet.split('.')[0] + "," + $destnet.split('.')[1]
                                        #else
                                                #set val121=$val121 + "," + $destnet.split('.')[0]
                                        #end if
                                        #
                                        #set val121=$val121 + "," + $gateway.replace('.', ',')
                                #end for

                option classless-routes $val121
                        #end if
        }
                #end if
        #end for
}

Obviously, there are likely bugs in this script, and I’m only using it on a couple of boxes in my lab network, so feel free to point out any issues in the comments and I’ll update the above accordingly.

A former roommate of mine once setup such a beast using a 12-port SATA card which ended up delivering a whopping 1 MBps of write speed in a RAID 5 configuration. I simply don’t have time to play around like that these days, so this is me trading capital for time.

The host machine is a Sun Fire X4200M2 server with an internal RAID10, running a RHEL 5.3 XEN installation. None of the services currently running on this box are critical, which means I can take them down for an hour at the end of the day without trouble, provided I can get them back up again. I also have the (Memorial Day) weekend to get the new JBOD up and running on this box.

After it’s up, however, I will be hosting important business-ey things on various virtual machines using this JBOD: e-mail, website(s), internal wiki, NAS, along with primary kerberos, LDAP, cobbler, puppet on the internal RAID; so it’s fairly important that this get up and working, and be stable once it’s going…

The JBOD itself is a Sun StorageTek J4200 array with a single IO module and a PCIe SAS RAID card, running 6x 1TB SATA disks in (eventually) a RAID6 array. I’d like to play around with interesting things like redundant SATA multipathing, but I’m pretty new to the whole storage admin area, so I’m not going to be playing around with those things on *this* setup…

So apparently, someone was trying to take online courses, ordered the cheapest Dell with a CD—which happens to be running Ubuntu—she could find, and then couldn’t get online to her courses. So she withdrew from the University, and the Linux Lusers rushed in—talking about how dumb she was for not being able to slickly navigate Linux through customer support in a Windows-only world, and apparently, this degenerated into people harassing her on Facebook.

There are a couple takeaways to this for the world at large:

1. Facebook works fine on Ubuntu (or the student in question has gotten a different Dell).
2. If you aren’t raising your kid to be able to handle computers like a nerd, you are handicapping your children’s ability to prosper.

Obviously, the second is the controversial opinion. While the new imperialist geek overlords are kinder, gentler overlords than the robber barons of the past, technology is a big ugly mess. The de-facto reality this illustrates is that if you are attempting to live in a modernized country, but are unable to figure out how to purchase and use a computer, you are fucked. Those who cannot figure out how to scam Central Services to get online are destined to be crushed underfoot in the information revolution. It’s an ugly, brutal reality. Fortunately, when dealing with economy, reality is what you make of it. There are a couple points for the democratic wing of the new masters:

1. There is a contingent of raving lunatics who have decided to immigrate to Linux as their chosen nationality.
2. When you smirk at the clueless n00b, you are the sadistic prison guard tormenting the hapless inmate. By making your system difficult for others to use, you are actually hurting them—not only in terms of time and stress, but also in financially measurable ways.

But none of that works on the real issue of this story: What was it about the Ubuntu desktop as shipped with Dell that prevented her from going to school? If you haven’t already, find out why our OS didn’t work for her, publicize the problems, and fix them. If it’s a technical problem then it’s completely trivial to fix: we’re all geeks here. If it was a more mushy social reason—the bureaucratic pronouncements of overworked support staff at her Uni and ISP: you must use MS Word on Windows (because we won’t support anything else)—then that’s something we have traditionally sucked at, but something which community growth could address in an indirect way, and B2B schmoozing could address in a direct way. Remember, she’s not the only one going through these difficulties, she’s just the only one who’s difficulties were severe enough to warrant a newspaper article on it.

1. This post is incomplete/incorrect. What I’m doing now is having the daemon function call a script that looks like this:

   #!/bin/bash
   exec 1>&-
   exec 2>&-
   exec 3>&-
   nohup myPropApp & 2>&1 > thelog.txt

   That code was from another website who’s URL I lost, and I posted the solution below based on another, alternate method that I hadn’t tried but sounded simpler.

2. There are other options, like daemonize(1), setsid(1), and the bash builtin disown (which I had prematurely rejected as ksh-only).

Back when I was using Debian, one of the nicer things about it was their helper tool for startup scripts: start-stop-daemon. Particularly, it’s ability to daemonize any process with the -b flag. You notice how handy things like that end up being when you’ve got an in-house or otherwise proprietary app that can’t daemonize itself properly (e.g. Java-based services).

Somehow I’ve managed to get away with not having to write a script that daemonizes a normally-foreground process on an RH-based distribution yet, mainly because I’ve been using Debian almost exclusively for servers, and have only worked for tiny startups, where luxuries like init scripts are the last thing on anyones’ minds.

Everyone is familiar with the nohup & trick, but that still leaves it associated to a terminal, so after you log out, your terminal/ssh session will just hang because stdin is still open. As it turns out, you can close your standard in from bash first by redirecting your standard input from nil (e.g. someapp <&-), and that will let it just work.

Very sweet for writing initscripts.

So, the requirements are Linux and Windows desktop support, a desire to teach yourself Asterisk, Windows domains, and Cisco networking, and the ability to pass a Federal security check. Experience with open-source web software and Apache (e.g. Wordpress, Joomla!, etc.) is great, but not required.

As before, send your resumé to me.

You must be familiar with remote administration techniques, MySQL, apache, VCS, RPM-based distributions, (the basics). Familiarity with bind, dhcpd, ddns, and basic networking is also recommended (at the very least you should be able to figure it out without handholding).

If this still sounds like something you’d like to participate in, send your resume to me and I’ll forward it on to our HR people for processing. Act today and you’ll get your very own number!

Is he free to do so? Also yes.

Are you free to ignore him? Still yes.

Does His Chomskiness actually take the challenge and provide a better rebuttal to the underlying book than politely demanding Miguel STFU? Yep.

Oh, and here’s a patch that will let you do something cool with XEN 3.0.3:

--- network-bridge      2007-02-08 09:21:12.000000000 -0600
+++ network-vlans       2007-09-14 09:55:20.000000000 -0500
@@ -26,6 +26,7 @@
 # bridge     The bridge to use (default xenbr${vifnum}).
 # netdev     The interface to add to the bridge (default eth${vifnum}).
 # antispoof  Whether to use iptables to prevent spoofing (default no).
+# vlans      VLANs to add on top of the bridge
 #
 # Internal Vars:
 # pdev="p${netdev}"
@@ -64,18 +65,27 @@
 bridge=${bridge:-xenbr${vifnum}}
 netdev=${netdev:-eth${vifnum}}
 antispoof=${antispoof:-no}
+vlans=$(echo $vlans | sed -e 's/,/ /g')

 pdev="p${netdev}"
 vdev="veth${vifnum}"
 vif0="vif0.${vifnum}"

 get_ip_info() {
-    addr_pfx=`ip addr show dev $1 | egrep '^ *inet' | sed -e 's/ *inet //' -e 's/ .*//'`
+    addr_pfx=`ip addr show dev $1 | sed -n 's/^ *inet \(.*\) [^ ]*$/\1/p'`
     gateway=`ip route show dev $1 | fgrep default | sed 's/default via //'`
 }
+
+is_bonding() {
+    [ -f "/sys/class/net/$1/bonding/slaves" ]
+}
+
+is_ifup() {
+    ip link show dev $1 | awk '{ exit $3 !~ /[< ,]UP[,>]/ }'
+}

 do_ifup() {
-    if ! ifup $1 ; then
+    if ! ifup $1 || ! is_ifup $1 ; then
         if [ ${addr_pfx} ] ; then
             # use the info from get_ip_info()
             ip addr flush $1
@@ -206,8 +216,8 @@
        mac=`ip link show ${netdev} | grep 'link\/ether' | sed -e 's/.*ether \(..:..:..:..:..:..\).*/\1/'`
        preiftransfer ${netdev}
        transfer_addrs ${netdev} ${vdev}
-       if ! ifdown ${netdev}; then
-           # If ifdown fails, remember the IP details.
+       if is_bonding ${netdev} || ! ifdown ${netdev}; then
+           # Remember the IP details if necessary.
            get_ip_info ${netdev}
            ip link set ${netdev} down
            ip addr flush ${netdev}
@@ -223,6 +233,18 @@
        add_to_bridge  ${bridge} ${vif0}
        add_to_bridge2 ${bridge} ${pdev}
        do_ifup ${netdev}
+
+       if [ -n "$vlans" ]; then
+               vconfig set_name_type VLAN_PLUS_VID_NO_PAD
+
+               for vlan in $vlans; do
+                       create_bridge xenbr${vlan}
+
+                       vconfig add ${bridge} ${vlan}
+                       setup_bridge_port vlan${vlan}
+                       add_to_bridge xenbr${vlan} vlan${vlan}
+               done
+       fi
     else
        # old style without ${vdev}
        transfer_addrs  ${netdev} ${bridge}
@@ -262,6 +284,20 @@
        ip link set ${netdev} name ${vdev}
        ip link set ${pdev} name ${netdev}
        do_ifup ${netdev}
+
+       if [ -n "$vlans" ]; then
+               for vlan in $vlans; do
+                       if [ -n `ip link show vlan${vlan} | grep '${bridge}\:'` ]; then
+                               ip link delif ${bridge} xenbr${vlan}
+                               ip link set ${bridge} down
+
+                               ip link set vlan${vlan} down
+                               vconfig rem ${bridge} ${vlan}
+                       fi
+               done
+
+               vconfig set_name_type DEV_PLUS_VID_NO_PAD
+       fi
     else
        transfer_routes ${bridge} ${netdev}
        ip link set ${bridge} down

It may be buggy, since I haven’t tested it in production. What it does is this: allows you to run an 802.1Q trunk into your XEN server, then put your virtual machines on any VLAN you want with a couple configuration stanzas.

So, your xend-config.sxp will have:

(network-script 'network-vlans netdev=eth0 vlans=8,9,10,11,13,121,14,15')

Which translates to “create bridges for VLAN 8, 9, 11, 13, 121, 14, and 15 with a xenbr prefix”. Then you set your DomU vif stanza to be “bridge=xenbr13” and bam! your DomU exists on the VLAN13. The primary limitation of this is that it keeps your Dom0 on the untagged/native VLAN, which isn’t best practice.

The stack of modules a packet traverses to get to a DomU will look like this (with relevant modules):

[network] -->
dom0: peth0 (dev) -->
dom0: xenbr0 (bridge) -->
dom0: vlan13 (dot1q attached to xenbr0) -->
dom0: xenbr13 (bridge) -->
dom0: vifX.0 (netloop) -->
domU: xen0 (xennet)