2009-07-17

Distributing Static Routes with DHCP

I’m set­ting up an iso­lated net­work for peo­ple to test inter­nal appli­ca­tions on, since the devel­op­ers all have Sun work­sta­tions with a dual-port Gigabit NIC on the moth­er­board, and we’ve got a bunch of older net­work equip­ment that we haven’t got­ten around to eBay­ing yet. What I’m doing is link­ing the sec­ond NICs together with some vir­tual machines and the older net­work equip­ment to cre­ate a sep­a­rate devel­op­ment network.

The devel­op­ment net­work is a full Layer-3 net­work run­ning an IGP between mul­ti­ple nodes with attached client boxes. This allows me to play around with a decent lab net­work, and pro­vides devel­op­ers with a way to dis­cover that Linux sets the TTL of mul­ti­cast pack­ets to “1” well before they are called to explain why their appli­ca­tion didn’t work even after loads of test­ing, spend 8 hours play­ing head-desk, and finally start ques­tion­ing me about fire­walls on our inter­nal net­work, forc­ing me to claw it out of them that they are dri­ving mul­ti­cast with­out a license and explain how to use tcpdump.

Not that I’ve had to do that a dozen times now, or any­thing…
Read the rest of this entry »

Be the second to comment on this...


2009-05-23

My First JBOD: Introduction

This is me set­ting up a JBOD for use by one or more XEN hosts, using pro­fes­sional hard­ware. It’s not a hack, not throw­ing a shit­load of dri­ves into a PC with some “pro­sumer” SATA RAID cards that require you spend weeks fuss­ing with dri­vers and firmware to get even a min­i­mal write per­for­mance out of their under­pow­ered hard­ware RAID.

A for­mer room­mate of mine once setup such a beast using a 12-port SATA card which ended up deliv­er­ing a whop­ping 1 MBps of write speed in a RAID 5 con­fig­u­ra­tion. I sim­ply don’t have time to play around like that these days, so this is me trad­ing cap­i­tal for time.

The host machine is a Sun Fire X4200M2 server with an inter­nal RAID10, run­ning a RHEL 5.3 XEN instal­la­tion. None of the ser­vices cur­rently run­ning on this box are crit­i­cal, which means I can take them down for an hour at the end of the day with­out trou­ble, pro­vided I can get them back up again. I also have the (Memorial Day) week­end to get the new JBOD up and run­ning on this box.

After it’s up, how­ever, I will be host­ing impor­tant business-ey things on var­i­ous vir­tual machines using this JBOD: e-mail, website(s), inter­nal wiki, NAS, along with pri­mary ker­beros, LDAP, cob­bler, pup­pet on the inter­nal RAID; so it’s fairly impor­tant that this get up and work­ing, and be sta­ble once it’s going…

The JBOD itself is a Sun StorageTek J4200 array with a sin­gle IO mod­ule and a PCIe SAS RAID card, run­ning 6x 1TB SATA disks in (even­tu­ally) a RAID6 array. I’d like to play around with inter­est­ing things like redun­dant SATA mul­ti­pathing, but I’m pretty new to the whole stor­age admin area, so I’m not going to be play­ing around with those things on *this* setup…

Comment on this...


2009-01-17

Ubuntu Ruined My Life

[There’s a whole bunch of mean­der­ing aca­d­e­mic pon­tif­i­cat­ing and me tak­ing myself too seri­ously. About two thirds of the way down it gets really good, though. I promise. Also, the woman is now online and back in school. –JC]

So appar­ently, some­one was try­ing to take online courses, ordered the cheap­est Dell with a CD — which hap­pens to be run­ning Ubuntu — she could find, and then couldn’t get online to her courses. So she with­drew from the University, and the Linux Lusers rushed in — talk­ing about how dumb she was for not being able to slickly nav­i­gate Linux through cus­tomer sup­port in a Windows-only world, and appar­ently, this degen­er­ated into peo­ple harass­ing her on Facebook.

There are a cou­ple take­aways to this for the world at large:

  1. Facebook works fine on Ubuntu (or the stu­dent in ques­tion has got­ten a dif­fer­ent Dell).
  2. If you aren’t rais­ing your kid to be able to han­dle com­put­ers like a nerd, you are hand­i­cap­ping your children’s abil­ity to prosper.

Obviously, the sec­ond is the con­tro­ver­sial opin­ion. While the new impe­ri­al­ist geek over­lords are kinder, gen­tler over­lords than the rob­ber barons of the past, tech­nol­ogy is a big ugly mess. The de-facto real­ity this illus­trates is that if you are attempt­ing to live in a mod­ern­ized coun­try, but are unable to fig­ure out how to pur­chase and use a com­puter, you are fucked. Those who can­not fig­ure out how to scam Central Services to get online are des­tined to be crushed under­foot in the infor­ma­tion rev­o­lu­tion. It’s an ugly, bru­tal real­ity. Fortunately, when deal­ing with econ­omy, real­ity is what you make of it. There are a cou­ple points for the demo­c­ra­tic wing of the new masters:

  1. There is a con­tin­gent of rav­ing lunatics who have decided to immi­grate to Linux as their cho­sen nationality.
  2. When you smirk at the clue­less n00b, you are the sadis­tic prison guard tor­ment­ing the hap­less inmate. By mak­ing your sys­tem dif­fi­cult for oth­ers to use, you are actu­ally hurt­ing them — not only in terms of time and stress, but also in finan­cially mea­sur­able ways.

But none of that works on the real issue of this story: What was it about the Ubuntu desk­top as shipped with Dell that pre­vented her from going to school? If you haven’t already, find out why our OS didn’t work for her, pub­li­cize the prob­lems, and fix them. If it’s a tech­ni­cal prob­lem then it’s com­pletely triv­ial to fix: we’re all geeks here. If it was a more mushy social rea­son — the bureau­cratic pro­nounce­ments of over­worked sup­port staff at her Uni and ISP: you must use MS Word on Windows (because we won’t sup­port any­thing else)—then that’s some­thing we have tra­di­tion­ally sucked at, but some­thing which com­mu­nity growth could address in an indi­rect way, and B2B schmooz­ing could address in a direct way. Remember, she’s not the only one going through these dif­fi­cul­ties, she’s just the only one who’s dif­fi­cul­ties were severe enough to war­rant a news­pa­per arti­cle on it.

Comment on this...


2008-10-16

Daemonizing Processes

Update: Commenters have pointed out a few things:

  1. This post is incomplete/incorrect. What I’m doing now is hav­ing the daemon func­tion call a script that looks like this:
    #!/bin/bash
    exec 1>&-
    exec 2>&-
    exec 3>&-
    nohup myPropApp & 2>&1 > thelog.txt

    That code was from another web­site who’s URL I lost, and I posted the solu­tion below based on another, alter­nate method that I hadn’t tried but sounded simpler.

  2. There are other options, like daemonize(1), setsid(1), and the bash builtin disown (which I had pre­ma­turely rejected as ksh-only).

Back when I was using Debian, one of the nicer things about it was their helper tool for startup scripts: start-stop-daemon. Particularly, it’s abil­ity to dae­mo­nize any process with the -b flag. You notice how handy things like that end up being when you’ve got an in-house or oth­er­wise pro­pri­etary app that can’t dae­mo­nize itself prop­erly (e.g. Java-based services).

Somehow I’ve man­aged to get away with not hav­ing to write a script that dae­mo­nizes a normally-foreground process on an RH-based dis­tri­b­u­tion yet, mainly because I’ve been using Debian almost exclu­sively for servers, and have only worked for tiny star­tups, where lux­u­ries like init scripts are the last thing on any­ones’ minds.

Everyone is famil­iar with the nohup & trick, but that still leaves it asso­ci­ated to a ter­mi­nal, so after you log out, your terminal/ssh ses­sion will just hang because stdin is still open. As it turns out, you can close your stan­dard in from bash first by redi­rect­ing your stan­dard input from nil (e.g. someapp <&-), and that will let it just work.

Very sweet for writ­ing initscripts.

Comment on this...


2008-07-04

More Help Wanted

As it turns out I have need for another Systems Administrator, this time in Washington, DC. This job is for a local admin­is­tra­tor to han­dle the day-to-day sup­port and activ­i­ties in the Washington office (com­plete with AD domain, Asterisk server, NAS, and a dozen users), as well as the four branch loca­tions in the DC Metro area and (future) dat­a­cen­ter while work­ing together via IM, mail, and phone with the exist­ing tech team in Chicago to plan and imple­ment improve­ments, and resolve prob­lems. The tech­no­log­i­cal envi­ron­ment is 80% Windows, but the remain­ing 20% is RHEL5; the branch loca­tions are 100% RHEL5.

So, the require­ments are Linux and Windows desk­top sup­port, a desire to teach your­self Asterisk, Windows domains, and Cisco net­work­ing, and the abil­ity to pass a Federal secu­rity check. Experience with open-source web soft­ware and Apache (e.g. Wordpress, Joomla!, etc.) is great, but not required.

As before, send your resumé to me.

Be the first to comment on this...


2008-06-22

Help Wanted

I’m look­ing to hire a Linux Administrator in for a posi­tion in down­town Chicago. It’s a high-demand, high-stress envi­ron­ment with lots of things going on at any one time: We play with high-end sun servers on an inter­na­tional pri­vate net­work, use Amazon EC2, and have a slew of Asterisk servers form­ing the joints of a wide-area VoIP infra­struc­ture. Success and fail­ure is often mea­sured in terms of mil­lisec­onds. On the down­side, we also do Windows, must sup­port the desk­top users (most desk­tops are Linux, though), and the com­pany isn’t large enough to jus­tify a divi­sion of labor yet.

You must be famil­iar with remote admin­is­tra­tion tech­niques, MySQL, apache, VCS, RPM-based dis­tri­b­u­tions, (the basics). Familiarity with bind, dhcpd, ddns, and basic net­work­ing is also rec­om­mended (at the very least you should be able to fig­ure it out with­out handholding).

If this still sounds like some­thing you’d like to par­tic­i­pate in, send your resume to me and I’ll for­ward it on to our HR peo­ple for pro­cess­ing. Act today and you’ll get your very own number!

Comment on this...


2007-10-01

Xen and The Art of Free Speech

Aside from the laugh­able idea of “mil­i­tantly” sup­port­ing any­thing with a blog post, Miguel sim­ply noted that these peo­ple exist, have writ­ten a book, and will be doing the speaking-tour-thing near him. Does he agree with the con­tents? (shakes eight-ball) Signs point to Yes.

Is he free to do so? Also yes.

Are you free to ignore him? Still yes.

Does His Chomskiness actu­ally take the chal­lenge and pro­vide a bet­ter rebut­tal to the under­ly­ing book than politely demand­ing Miguel STFU? Yep.

Oh, and here’s a patch that will let you do some­thing cool with XEN 3.0.3:

--- network-bridge      2007-02-08 09:21:12.000000000 -0600
+++ network-vlans       2007-09-14 09:55:20.000000000 -0500
@@ -26,6 +26,7 @@
 # bridge     The bridge to use (default xenbr${vifnum}).
 # netdev     The interface to add to the bridge (default eth${vifnum}).
 # antispoof  Whether to use iptables to prevent spoofing (default no).
+# vlans      VLANs to add on top of the bridge
 #
 # Internal Vars:
 # pdev="p${netdev}"
@@ -64,18 +65,27 @@
 bridge=${bridge:-xenbr${vifnum}}
 netdev=${netdev:-eth${vifnum}}
 antispoof=${antispoof:-no}
+vlans=$(echo $vlans | sed -e 's/,/ /g')

 pdev="p${netdev}"
 vdev="veth${vifnum}"
 vif0="vif0.${vifnum}"

 get_ip_info() {
-    addr_pfx=`ip addr show dev $1 | egrep '^ *inet' | sed -e 's/ *inet //' -e 's/ .*//'`
+    addr_pfx=`ip addr show dev $1 | sed -n 's/^ *inet \(.*\) [^ ]*$/\1/p'`
     gateway=`ip route show dev $1 | fgrep default | sed 's/default via //'`
 }
+
+is_bonding() {
+    [ -f "/sys/class/net/$1/bonding/slaves" ]
+}
+
+is_ifup() {
+    ip link show dev $1 | awk '{ exit $3 !~ /[< ,]UP[,>]/ }'
+}

 do_ifup() {
-    if ! ifup $1 ; then
+    if ! ifup $1 || ! is_ifup $1 ; then
         if [ ${addr_pfx} ] ; then
             # use the info from get_ip_info()
             ip addr flush $1
@@ -206,8 +216,8 @@
        mac=`ip link show ${netdev} | grep 'link\/ether' | sed -e 's/.*ether \(..:..:..:..:..:..\).*/\1/'`
        preiftransfer ${netdev}
        transfer_addrs ${netdev} ${vdev}
-       if ! ifdown ${netdev}; then
-           # If ifdown fails, remember the IP details.
+       if is_bonding ${netdev} || ! ifdown ${netdev}; then
+           # Remember the IP details if necessary.
            get_ip_info ${netdev}
            ip link set ${netdev} down
            ip addr flush ${netdev}
@@ -223,6 +233,18 @@
        add_to_bridge  ${bridge} ${vif0}
        add_to_bridge2 ${bridge} ${pdev}
        do_ifup ${netdev}
+
+       if [ -n "$vlans" ]; then
+               vconfig set_name_type VLAN_PLUS_VID_NO_PAD
+
+               for vlan in $vlans; do
+                       create_bridge xenbr${vlan}
+
+                       vconfig add ${bridge} ${vlan}
+                       setup_bridge_port vlan${vlan}
+                       add_to_bridge xenbr${vlan} vlan${vlan}
+               done
+       fi
     else
        # old style without ${vdev}
        transfer_addrs  ${netdev} ${bridge}
@@ -262,6 +284,20 @@
        ip link set ${netdev} name ${vdev}
        ip link set ${pdev} name ${netdev}
        do_ifup ${netdev}
+
+       if [ -n "$vlans" ]; then
+               for vlan in $vlans; do
+                       if [ -n `ip link show vlan${vlan} | grep '${bridge}\:'` ]; then
+                               ip link delif ${bridge} xenbr${vlan}
+                               ip link set ${bridge} down
+
+                               ip link set vlan${vlan} down
+                               vconfig rem ${bridge} ${vlan}
+                       fi
+               done
+
+               vconfig set_name_type DEV_PLUS_VID_NO_PAD
+       fi
     else
        transfer_routes ${bridge} ${netdev}
        ip link set ${bridge} down

It may be buggy, since I haven’t tested it in pro­duc­tion. What it does is this: allows you to run an 802.1Q trunk into your XEN server, then put your vir­tual machines on any VLAN you want with a cou­ple con­fig­u­ra­tion stanzas.

So, your xend-config.sxp will have:

(network-script 'network-vlans netdev=eth0 vlans=8,9,10,11,13,121,14,15')

Which trans­lates to “cre­ate bridges for VLAN 8, 9, 11, 13, 121, 14, and 15 with a xenbr pre­fix”. Then you set your DomU vif stanza to be “bridge=xenbr13” and bam! your DomU exists on the VLAN13. The pri­mary lim­i­ta­tion of this is that it keeps your Dom0 on the untagged/native VLAN, which isn’t best practice.

The stack of mod­ules a packet tra­verses to get to a DomU will look like this (with rel­e­vant modules):

[network] -->
dom0: peth0 (dev) -->
dom0: xenbr0 (bridge) -->
dom0: vlan13 (dot1q attached to xenbr0) -->
dom0: xenbr13 (bridge) -->
dom0: vifX.0 (netloop) -->
domU: xen0 (xennet)

Comment on this...