One of the recurring problems I have with Zenoss is fixing the swap threshold issue. Basically, if your swap space is less than 1G, you’re stuck with an alarm informing you that there’s less than 1G of swap total. The options are to hack it to increase the threshold (by decreasing the minimum-free threshold), or to make the alert use a percentage of the total.
Posting it here since it’s the second time I’ve had to find this…
One of the bad ideas currently infecting companies in the technology field is the LivePerson “chat with a support person now” thing. This is a bad idea for multiple reasons:
Case in point: Limelight Networks. The website lists about 15 “Services” they offer, and none of them are immediately obvious and I don’t want to spend the next hour parsing their marketease trying to find the specific niche product I’m looking for. So I (like a fool) click the stupid chat thing obscuring their page. This started at 8:30am and went on until 9:30am.
Jon: Yes what company or site are you from so I can better help you?
You: [redacted]
Jon: Thank you for waiting. I’ll be with you in just a moment.
Jon: I’m sorry for the delay. I’ll be right with you.
Jon: I’ll be right with you.
You: Why don’t you have a salesperson contact me and just ask me if our website is, in fact, a blank page?
Jon: I’m sorry for the delay. I’ll be right with you.
Jon: Thank you for waiting. I’ll be with you in just a moment.
Jon: I’ll be right with you.
Jon: I’ll be right with you.
Jon: Thank you for waiting. I’ll be with you in just a moment.
You: So what’s the point of this chat, exactly? For me to wait on iHold for an hour while canned messages about your imminent reply scroll past?
You: Until I get bored and find another vendor?
After another couple minutes I just closed the window, and now I’m at Akamai’s site, looking at their “solutions finder” — which is what I was looking for on Limelight’s website to begin with.
I’ve spent the last couple weeks moving off of my existing server(s) and into the cloud. Previously, I had been using my own Zimbra server, own SVN/trac install, and websites, albeit virtualized on a shared XEN server. The physical server all this was running on was some ancient second-hand single-core i386 Dell poweredge which never had enough RAM, cyles or bandwidth.
For this, I paid a friend of mine $30/mo. Recently, the fourth person in our arrangement dropped out and so our costs went up to $40/mo. Now, I had 768Mb worth of memory on the two virtual machines I had, of which I was only actually using one.
So I was paying $40/mo for a single VM instance I ran SCM and my website off of, and my e-mail. That’s dumb, since you can use private repos on Github for $7/mo, use Gmail for free (all things equal, webmail is webmail), and get a private VM instance on Linode for $17/mo.
So that’s what I did: I cut my costs in half over the self-hosted solution by putting shit online.
Now, if me and my friends had kicked in a lot more $ and gotten a real server, and split it that 1RU up more aggressively, then it would have been cheaper to do that ourselves. But nobody cared enough about that to make it work, so putting it elsewhere is cheaper.
Which is a generalized conclusion I’m willing to draw: if nobody cares, it’s cheaper to pay someone to do it than to muddle through yourself. If someone does care, then it’s invariably going to be cheaper to DIY.
I’m setting up an isolated network for people to test internal applications on, since the developers all have Sun workstations with a dual-port Gigabit NIC on the motherboard, and we’ve got a bunch of older network equipment that we haven’t gotten around to eBaying yet. What I’m doing is linking the second NICs together with some virtual machines and the older network equipment to create a separate development network.
The development network is a full Layer-3 network running an IGP between multiple nodes with attached client boxes. This allows me to play around with a decent lab network, and provides developers with a way to discover that Linux sets the TTL of multicast packets to “1” well before they are called to explain why their application didn’t work even after loads of testing, spend 8 hours playing head-desk, and finally start questioning me about firewalls on our internal network, forcing me to claw it out of them that they are driving multicast without a license and explain how to use tcpdump.
Not that I’ve had to do that a dozen times now, or anything…
Read the rest of this entry »
After unpacking, racking, and mounting the JBOD, I waited until the weekend had started before powering down the server and installing the RAID card. Connected it all up, rebooted into the Adaptec BIOS, and configured the 6x 1TB drives into a RAID6 array. After that, I installed the RAID StorageManager off of Sun’s website, and then the “Common Array Manager” software. CAM is supposed to provide a web GUI to an organization’s worth of Sun JBODs, so you can update JBOD firmware and query status and whatnot from a single interface. There’s client and server bits written in Java that run on the various boxes, so the data path was going to look like this:
JBOD -> XEN dom0 running remote proxy tool -> XEN domU running web GUI
I say “was going” and “supposed to” because all the remote proxy tool in CAM ended up doing was consistently triggering a kernel panic in the aacraid driver whenever it’s detection code fired up.
Take a long drag off the irony of driver and firmware issues, and download the latest-n-greatest aacraid driver and firmware from Intel via Sun, and update. Same results. Repeat in various configurations, and before throwing in the towel, get a basic dump and file a bug. I didn’t put any more serious thought into debugging it simply because this whole thing has to be up and running yesterday, and the last time I asked for documentation on the topic, I was rebuffed with a variant of this classic: “If you were smart enough to debug the kernel, you wouldn’t need documentation on how to debug the kernel.”
Take a moment to stand in awe of the massive poisonous cobaggery involved in that statement being offered to someone who wants to help fix a crasher. I’ll wait.
That kind of shit would never fly in any GNOME venue, which is why GNOME kicks so much ass.
Update: The cobaggery about kernel development did not come from Sun or any representative of any company involved in open-source, and was unrelated to this situation at all. I relate it simply as it pertains to debugging kernel issues, and why I don’t do it.
This is me setting up a JBOD for use by one or more XEN hosts, using professional hardware. It’s not a hack, not throwing a shitload of drives into a PC with some “prosumer” SATA RAID cards that require you spend weeks fussing with drivers and firmware to get even a minimal write performance out of their underpowered hardware RAID.
A former roommate of mine once setup such a beast using a 12-port SATA card which ended up delivering a whopping 1 MBps of write speed in a RAID 5 configuration. I simply don’t have time to play around like that these days, so this is me trading capital for time.
The host machine is a Sun Fire X4200M2 server with an internal RAID10, running a RHEL 5.3 XEN installation. None of the services currently running on this box are critical, which means I can take them down for an hour at the end of the day without trouble, provided I can get them back up again. I also have the (Memorial Day) weekend to get the new JBOD up and running on this box.
After it’s up, however, I will be hosting important business-ey things on various virtual machines using this JBOD: e-mail, website(s), internal wiki, NAS, along with primary kerberos, LDAP, cobbler, puppet on the internal RAID; so it’s fairly important that this get up and working, and be stable once it’s going…
The JBOD itself is a Sun StorageTek J4200 array with a single IO module and a PCIe SAS RAID card, running 6x 1TB SATA disks in (eventually) a RAID6 array. I’d like to play around with interesting things like redundant SATA multipathing, but I’m pretty new to the whole storage admin area, so I’m not going to be playing around with those things on *this* setup…
Latest on the “done” pile are Rule The Freakin’ Markets and IS-IS Network Design Solutions. Summaries/reviews of both are up.
Update: Commenters have pointed out a few things:
daemon function call a script that looks like this:
#!/bin/bash exec 1>&- exec 2>&- exec 3>&- nohup myPropApp & 2>&1 > thelog.txt
That code was from another website who’s URL I lost, and I posted the solution below based on another, alternate method that I hadn’t tried but sounded simpler.
daemonize(1), setsid(1), and the bash builtin disown (which I had prematurely rejected as ksh-only).Back when I was using Debian, one of the nicer things about it was their helper tool for startup scripts: start-stop-daemon. Particularly, it’s ability to daemonize any process with the -b flag. You notice how handy things like that end up being when you’ve got an in-house or otherwise proprietary app that can’t daemonize itself properly (e.g. Java-based services).
Somehow I’ve managed to get away with not having to write a script that daemonizes a normally-foreground process on an RH-based distribution yet, mainly because I’ve been using Debian almost exclusively for servers, and have only worked for tiny startups, where luxuries like init scripts are the last thing on anyones’ minds.
Everyone is familiar with the nohup & trick, but that still leaves it associated to a terminal, so after you log out, your terminal/ssh session will just hang because stdin is still open. As it turns out, you can close your standard in from bash first by redirecting your standard input from nil (e.g. someapp <&-), and that will let it just work.
Very sweet for writing initscripts.
As it turns out I have need for another Systems Administrator, this time in Washington, DC. This job is for a local administrator to handle the day-to-day support and activities in the Washington office (complete with AD domain, Asterisk server, NAS, and a dozen users), as well as the four branch locations in the DC Metro area and (future) datacenter while working together via IM, mail, and phone with the existing tech team in Chicago to plan and implement improvements, and resolve problems. The technological environment is 80% Windows, but the remaining 20% is RHEL5; the branch locations are 100% RHEL5.
So, the requirements are Linux and Windows desktop support, a desire to teach yourself Asterisk, Windows domains, and Cisco networking, and the ability to pass a Federal security check. Experience with open-source web software and Apache (e.g. Wordpress, Joomla!, etc.) is great, but not required.
As before, send your resumé to me.
I’m looking to hire a Linux Administrator in for a position in downtown Chicago. It’s a high-demand, high-stress environment with lots of things going on at any one time: We play with high-end sun servers on an international private network, use Amazon EC2, and have a slew of Asterisk servers forming the joints of a wide-area VoIP infrastructure. Success and failure is often measured in terms of milliseconds. On the downside, we also do Windows, must support the desktop users (most desktops are Linux, though), and the company isn’t large enough to justify a division of labor yet.
You must be familiar with remote administration techniques, MySQL, apache, VCS, RPM-based distributions, (the basics). Familiarity with bind, dhcpd, ddns, and basic networking is also recommended (at the very least you should be able to figure it out without handholding).
If this still sounds like something you’d like to participate in, send your resume to me and I’ll forward it on to our HR people for processing. Act today and you’ll get your very own number!
Be the first to comment on this...