Posting it here since it’s the second time I’ve had to find this…

1. It’s a gigantic floating piece of garbage distracting me from whatever it is I’m trying to learn about your company or it’s products.
2. The representatives on the other side are idiots who wouldn’t know what customer service is were it [insert your preferred-gendered joke here].

Case in point: Limelight Networks. The website lists about 15 “Services” they offer, and none of them are immediately obvious and I don’t want to spend the next hour parsing their marketease trying to find the specific niche product I’m looking for. So I (like a fool) click the stupid chat thing obscuring their page. This started at 8:30am and went on until 9:30am.

Jon: Yes what company or site are you from so I can better help you?
You: [redacted]
Jon: Thank you for waiting. I’ll be with you in just a moment.
Jon: I’m sorry for the delay. I’ll be right with you.
Jon: I’ll be right with you.
You: Why don’t you have a salesperson contact me and just ask me if our website is, in fact, a blank page?
Jon: I’m sorry for the delay. I’ll be right with you.
Jon: Thank you for waiting. I’ll be with you in just a moment.
Jon: I’ll be right with you.
Jon: I’ll be right with you.
Jon: Thank you for waiting. I’ll be with you in just a moment.
You: So what’s the point of this chat, exactly? For me to wait on iHold for an hour while canned messages about your imminent reply scroll past?
You: Until I get bored and find another vendor?

After another couple minutes I just closed the window, and now I’m at Akamai’s site, looking at their “solutions finder” — which is what I was looking for on Limelight’s website to begin with.

For this, I paid a friend of mine $30/mo. Recently, the fourth person in our arrangement dropped out and so our costs went up to $40/mo. Now, I had 768Mb worth of memory on the two virtual machines I had, of which I was only actually using one.

So I was paying $40/mo for a single VM instance I ran SCM and my website off of, and my e-mail. That’s dumb, since you can use private repos on Github for $7/mo, use Gmail for free (all things equal, webmail is webmail), and get a private VM instance on Linode for $17/mo.

So that’s what I did: I cut my costs in half over the self-hosted solution by putting shit online.

Now, if me and my friends had kicked in a lot more $ and gotten a real server, and split it that 1RU up more aggressively, then it would have been cheaper to do that ourselves. But nobody cared enough about that to make it work, so putting it elsewhere is cheaper.

Which is a generalized conclusion I’m willing to draw: if nobody cares, it’s cheaper to pay someone to do it than to muddle through yourself. If someone does care, then it’s invariably going to be cheaper to DIY.

The development network is a full Layer-3 network running an IGP between multiple nodes with attached client boxes. This allows me to play around with a decent lab network, and provides developers with a way to discover that Linux sets the TTL of multicast packets to “1” well before they are called to explain why their application didn’t work even after loads of testing, spend 8 hours playing head-desk, and finally start questioning me about firewalls on our internal network, forcing me to claw it out of them that they are driving multicast without a license and explain how to use tcpdump.

Not that I’ve had to do that a dozen times now, or anything…

This means I have to configure static routes on the developer workstations so they can access things in the lab outside their local subnet. You start off by configuring static routes in your distro’s chosen format (this is RHEL5 at work, so it’s /etc/sysconfig/network-scripts/route-ethX), and then you step it up a notch by writing scripts to distribute these files, then start using rgang or func, and start thinking about using your systems programming tool to distribute the routes. And then you smack your forehead and figure out that this is all stupid: there is already an IETF standard way to distribute network configuration which you should be using: DHCP.

There’s even DHCP option 121, which provides a way to distribute CIDR information (modern static routes) to clients. Unfortunately this standard option isn’t supported out of the box on modern dhclient or ISC dhcpd, so you need to configure it and script it in.

First, on the client, /etc/dhclient-exit-hooks

#!/bin/bash
#
# /etc/dhclient-exit-hooks
#
# This file is called from /sbin/dhclient-script after a DHCP run.
#

#
# parse_option_121:
# @argv: the array contents of DHCP option 121, separated by spaces.
# @returns: a colon-separated list of arguments to pass to /sbin/ip route
#
function parse_option_121() {
        result=""

        while [ $# -ne 0 ]; do
                mask=$1
                shift

                # Is the destination a multicast group?
                if [ $1 -ge 224 -a $1 -lt 240 ]; then
                        multicast=1
                else
                        multicast=0
                fi

                # Parse the arguments into a CIDR net/mask string
                if [ $mask -gt 24 ]; then
                        destination="$1.$2.$3.$4/$mask"
                        shift; shift; shift; shift
                elif [ $mask -gt 16 ]; then
                        destination="$1.$2.$3.0/$mask"
                        shift; shift; shift
                elif [ $mask -gt 8 ]; then
                        destination="$1.$2.0.0/$mask"
                        shift; shift
                else
                        destination="$1.0.0.0/$mask"
                        shift
                fi

                # Read the gateway
                gateway="$1.$2.$3.$4"
                shift; shift; shift; shift

                # Multicast routing on Linux
                #  - If you set a next-hop address for a multicast group, this breaks with Cisco switches
                #  - If you simply leave it link-local and attach it to an interface, it works fine.
                if [ $multicast -eq 1 ]; then
                        temp_result="$destination dev $interface"
                else
                        temp_result="$destination via $gateway dev $interface"
                fi

                if [ -n "$result" ]; then
                        result="$result:$temp_result"
                else
                        result="$temp_result"
                fi
        done

        echo "$result"
}

function modify_routes() {
        action=$1
        route_list="$2"

        IFS=:
        for route in $route_list; do
                unset IFS
                /sbin/ip route $action $route
                IFS=:
        done
        unset IFS
}

if [ "$reason" = "BOUND" -o "$reason" = "REBOOT" -o "$reason" = "REBIND" -o "$reason" = "RENEW" ]; then
        # Delete old routes, if they exist
        if [ -n "$old_classless_routes" ]; then
                modify_routes delete "$(parse_option_121 $old_classless_routes)"
        fi

        # Add new routes, if they exist...
        if [ -n "$new_classless_routes" ]; then
                modify_routes add "$(parse_option_121 $new_classless_routes)"
        fi
fi

We use /etc/dhclient-exit-hooks because the RHEL5 dhclient-script only calls the up-hooks script on BOUND and REBOOT, so if you change your static routes on the server, your client won’t pick them up until the box reboots or the interface is otherwise cycled.

The obvious problem here is that it’s always deleting the old routes and adding the new routes in two stages, a worthwhile enhancement for this script is to diff the old and new routes and determine which ones actually need to be removed/added.

So that will not do anything at first, because dhclient doesn’t actually read option 121 until you tell it to. For that, you need to edit /etc/dhclient.conf, and tell it how to handle option 121 in a way that the script above can understand:

#
# dhclient.conf
#

option classless-routes code 121 = array of unsigned integer 8;
request;

This tells dhclient to read all options, parse option 121 into an array of numeric bytes, and provide that array as a space-separated string as the new_classless_routes and old_classless_routes variables.

So now we’ve gotten all that taken care of, we need to start distributing routes from the DHCP server. For that, you need to update your /etc/dhcpd.conf file:

#
# dhcpd.conf
#

option classless-routes code 121 = array of unsigned integer 8;

subnet 10.23.1.0 netmask 255.255.255.0 {
        [...]
        # Routes for 10.23.0.0/16 via 10.23.1.1, and 224.0.0.0/4 (all IP multicast) via same
        option classless-routes 16,10,23,10,23,1,1,4,224,10,23,1,1
        [...]
}

You can also put that option into a host stanza if you’re doing that. Finally, as I’m using cobbler, I wanted to be able to have the new “static-routes” interface option end up in my cobbler-managed DHCPd configuration. Here’s a bit of my template that puts that configuration option into the appropriate DHCP option:

#
# /etc/cobbler/dhcp.template
#

[...]

#for dhcp_tag in $dhcp_tags.keys()
group {
        #for mac in $dhcp_tags[$dhcp_tag].keys():
                #set iface = $dhcp_tags[$dhcp_tag][$mac]
                #if $iface.dns_name
        host $iface.dns_name {
                hardware ethernet $mac;
                        #if $iface.ip_address
                fixed-address $iface.dns_name;
                        #else
                ddns-hostname "${iface.dns_name.split('.')[0]}";
                        #end if
                        #if $iface.static_routes:
                                #set val121=""
                                #for routespec in $iface.static_routes:
                                        #set gateway=$routespec.split(':')[1]
                                        #set destcidr=$routespec.split(':')[0]
                                        #set destnet=$destcidr.split('/')[0]
                                        #set destmask=$destcidr.split('/')[1]
                                        #
                                        #if val121
                                                #set val121=$val121 + ",$destmask"
                                        #else
                                                #set val121=$destmask
                                        #end if
                                        #
                                        #if int($destmask) > 24
                                                #set val121=$val121 + "," + $destnet.replace('.', ',')
                                        #else if int($destmask) > 16
                                                #set val121=$val121 + "," + $destnet.split('.')[0] + "," + $destnet.split('.')[1] + "," + $destnet.split('.')[2]
                                        #else if int($destmask) > 8
                                                #set val121=$val121 + "," + $destnet.split('.')[0] + "," + $destnet.split('.')[1]
                                        #else
                                                #set val121=$val121 + "," + $destnet.split('.')[0]
                                        #end if
                                        #
                                        #set val121=$val121 + "," + $gateway.replace('.', ',')
                                #end for

                option classless-routes $val121
                        #end if
        }
                #end if
        #end for
}

Obviously, there are likely bugs in this script, and I’m only using it on a couple of boxes in my lab network, so feel free to point out any issues in the comments and I’ll update the above accordingly.

JBOD -> XEN dom0 running remote proxy tool -> XEN domU running web GUI

I say “was going” and “supposed to” because all the remote proxy tool in CAM ended up doing was consistently triggering a kernel panic in the aacraid driver whenever it’s detection code fired up.

Take a long drag off the irony of driver and firmware issues, and download the latest-n-greatest aacraid driver and firmware from Intel via Sun, and update. Same results. Repeat in various configurations, and before throwing in the towel, get a basic dump and file a bug. I didn’t put any more serious thought into debugging it simply because this whole thing has to be up and running yesterday, and the last time I asked for documentation on the topic, I was rebuffed with a variant of this classic: “If you were smart enough to debug the kernel, you wouldn’t need documentation on how to debug the kernel.”

Take a moment to stand in awe of the massive poisonous cobaggery involved in that statement being offered to someone who wants to help fix a crasher. I’ll wait.

That kind of shit would never fly in any GNOME venue, which is why GNOME kicks so much ass.

Update: The cobaggery about kernel development did not come from Sun or any representative of any company involved in open-source, and was unrelated to this situation at all. I relate it simply as it pertains to debugging kernel issues, and why I don’t do it.

A former roommate of mine once setup such a beast using a 12-port SATA card which ended up delivering a whopping 1 MBps of write speed in a RAID 5 configuration. I simply don’t have time to play around like that these days, so this is me trading capital for time.

The host machine is a Sun Fire X4200M2 server with an internal RAID10, running a RHEL 5.3 XEN installation. None of the services currently running on this box are critical, which means I can take them down for an hour at the end of the day without trouble, provided I can get them back up again. I also have the (Memorial Day) weekend to get the new JBOD up and running on this box.

After it’s up, however, I will be hosting important business-ey things on various virtual machines using this JBOD: e-mail, website(s), internal wiki, NAS, along with primary kerberos, LDAP, cobbler, puppet on the internal RAID; so it’s fairly important that this get up and working, and be stable once it’s going…

The JBOD itself is a Sun StorageTek J4200 array with a single IO module and a PCIe SAS RAID card, running 6x 1TB SATA disks in (eventually) a RAID6 array. I’d like to play around with interesting things like redundant SATA multipathing, but I’m pretty new to the whole storage admin area, so I’m not going to be playing around with those things on *this* setup…

1. This post is incomplete/incorrect. What I’m doing now is having the daemon function call a script that looks like this:

   #!/bin/bash
   exec 1>&-
   exec 2>&-
   exec 3>&-
   nohup myPropApp & 2>&1 > thelog.txt

   That code was from another website who’s URL I lost, and I posted the solution below based on another, alternate method that I hadn’t tried but sounded simpler.

2. There are other options, like daemonize(1), setsid(1), and the bash builtin disown (which I had prematurely rejected as ksh-only).

Back when I was using Debian, one of the nicer things about it was their helper tool for startup scripts: start-stop-daemon. Particularly, it’s ability to daemonize any process with the -b flag. You notice how handy things like that end up being when you’ve got an in-house or otherwise proprietary app that can’t daemonize itself properly (e.g. Java-based services).

Somehow I’ve managed to get away with not having to write a script that daemonizes a normally-foreground process on an RH-based distribution yet, mainly because I’ve been using Debian almost exclusively for servers, and have only worked for tiny startups, where luxuries like init scripts are the last thing on anyones’ minds.

Everyone is familiar with the nohup & trick, but that still leaves it associated to a terminal, so after you log out, your terminal/ssh session will just hang because stdin is still open. As it turns out, you can close your standard in from bash first by redirecting your standard input from nil (e.g. someapp <&-), and that will let it just work.

Very sweet for writing initscripts.

So, the requirements are Linux and Windows desktop support, a desire to teach yourself Asterisk, Windows domains, and Cisco networking, and the ability to pass a Federal security check. Experience with open-source web software and Apache (e.g. Wordpress, Joomla!, etc.) is great, but not required.

As before, send your resumé to me.

You must be familiar with remote administration techniques, MySQL, apache, VCS, RPM-based distributions, (the basics). Familiarity with bind, dhcpd, ddns, and basic networking is also recommended (at the very least you should be able to figure it out without handholding).

If this still sounds like something you’d like to participate in, send your resume to me and I’ll forward it on to our HR people for processing. Act today and you’ll get your very own number!