You can’t depend on your eyes when your imagination is out of focus. – Mark Twain

Rusty Skills

I need to cut holes into a 2-inch PCV pipe. Round or square, it doesn’t matter. Holes about the size of a quarter.

I need to cover those holes in a low-pressure air-tight way.

I need a way to lift or drop the covers via levers, cables or something else based on a key. Preferably a key or lever that travels no more than about a centimeter vertically when it is pressed.

The mechanism that raises and lowers the covers needs to be affixed to the PVC pipe.

My skills are rusty. I would really like to use occams razor here and keep the mechanism as simple as possible.

I was pondering trying to do something with bicycle brake cables and such, but what to affix the covers and hinge the action with…..

The quandry

I look forward to having a time and place to experiment again. As a younger programmer or admin, you are expected to be learning and taking time to improve as part of your tasks. As a more senior admin or programmer, you are expected to be proficient and expert in your field. That is good, but it can become a bit of a box when you would like to learn something new.

Right now I spend a lot of time on high-visbility and production things. This means it is not a time to play. One does not make an experiment of a multi-billion view website.

But it is always tempting to dabble in new tech and new paths. Digital sandboxing keeps the mind fresh.

It can have a bitter aftertaste though. My time for such things is precious and I want to use it wisely. It is frustrating as you work through doing something on a new tool that you know you can accomplish perfectly fine with another toolset. It becomes doubly so when they have some odd way of doing things.

I really think we will have mature technology when we figure out a way to keep users, and eventually creators (programmers, etc) from having to relearn the same things as part of an iteration of tools. How many word processors have you had to learn? I knew my first one inside and out and I bet you did too. By the third or fourth one, you just bang around menus until you find where they “hid” the option in their layout.

I could really do without that waste of my time.

I write like….

I am a hack writer. Some days, I hope I am better than your average hack writer. Anyway, let’s see how this meme goes.

I dumped in a story that I wrote where I was trying to emulate the humor of Douglas Adams:

I write like
Mario Puzo

I Write Like by Mémoires, Mac journal software. Analyze your writing!

Hrmm. I’m not so sure on that. So, I put in a story where I tried to emulate a bit of an encapsulated fairy tale in a larger story:

I write like
Daniel Defoe

I Write Like by Mémoires, Mac journal software. Analyze your writing!

By now, I’m curious how they score the results. The results are reliable for the same text, so it is not random. Let’s check a few things:

Let’s see how Mark Twain’s famous essay “On the Decay of the Art of Lying” scores. Mr. Mark Twain writes like:

I write like
H. P. Lovecraft

I Write Like by Mémoires, Mac journal software. Analyze your writing!

Well! They were contemporaries of each other. But I don’t think Cuthulu ever showed up, even masquerading as Huck Finn. Interesting score there.

Let’s try another famous text. How about the Gettysburg Address by Honest Abe? President Lincoln writes like:

I write like
H. P. Lovecraft

I Write Like by Mémoires, Mac journal software. Analyze your writing!

Whoah. Two Lovecrafts in a row. Maybe if the grammer is verbose, you score Lovecraft. So let’s think verbose and test that:

The english translation of Mein Kampf is about the most verbose thing I’ve ever read. I know James Joyce and Dostoyevsky are both high on the verbosity scale, but Hitler was long-winded even by german standards. Add to that the lengthening effects of translating German to English and you end up with a spectacular result. (There is one point where a sentence runs longer than a page and a half.) So tossing a chapter on the founding of the nazi party at this thing scores, Lovecraft, right?

I write like
George Orwell

I Write Like by Mémoires, Mac journal software. Analyze your writing!

That’s a surprise. Apparently Hitler writes like George Orwell. That does make an odd sort of sense.

Still. Something spooks me about Lovecraft scoring so often.

Let’s really test things. How about 25 paragraphs of Lorem Ipsum?

I write like
H. P. Lovecraft

I Write Like by Mémoires, Mac journal software. Analyze your writing!

Lovecraft again! So now I’m thinking “H.P. Lovecraft” is their default for “very verbose and no word context.”

So let’s toss something really off the wall then. Something that modern context slips on. How about Aristophanes’ 455 BC sexual comedy “Lysistrata”?

I write like
James Joyce

I Write Like by Mémoires, Mac journal software. Analyze your writing!

James Joyce?! Really? Well. Maybe the section about the greek chorus beating actors with symbolic penis clubs. That’s kind of James Joyce.

I am curious about the Joyce score though. He writes sentences similar in complexity to the generation of Twain and Lovecraft. Lets check how Joyce scores with his “A Portrait of the Artist as a Young Man.”

I write like
James Joyce

I Write Like by Mémoires, Mac journal software. Analyze your writing!

Well. There ya go. Maybe it does kind-of work after all. I think it’s relying heavily on sentence structure, and might get a bit distracted if you write in an older style.

Abe Lincoln as a Lovecraftian evil. Well. He did preside over the largest loss of life in his country and a resurgence of spiritualism….

Doing it wrong… part 1

At one point, while walking a person through one of the VMware provided sample scripts I found myself talking about the perl mistakes that VMware was commonly making and my frustration with them. It turns out, it makes an interesting micro-view of how I find VMware’s perl code and SDKs frustrating.

I know that there’s more than one way to do things. But some ways are really better than others. The perl community is no exception at recognizing this: There’s a great book out there on perl best practices. And a wonderful website out there (Perl::Critic) that will vet your code for many common problems and mistakes. And lots of experienced folks who are glad to help discuss why some things are done better “the Perl way.”

So, for this post I’m just going over the same sample script I was discussing with that client: “” (Its documentation is here and it is a part of the VIPERL distribution from VMware and is copyright to VMware.) The goal here is for me to just brain dump some of the usage errors I see in even the VMware sample code. Maybe we’ll tackle bigger issues in another post.

#1 Bad Namespace Usage

The first thing I find annoying with the VMware SDK as evidenced by the the toolkit is the mismatched namespace. In general in perl if I “use Foo” I expect my module to show up in the “Foo” namespace.

VMware pitches that out the window for no apparent reason. On line 15, “use AppUtil::HostUtil” actually populates two name spaces “Opts” and “Util.” Why? And why two? I expect a module I have imported to contain sub-modules yes. But I do not expect multiple global name spaces to be filled.

#2 Global references to other namespaces

Which beings me to the next frustration. Using other name spaces directly.

In perl it is possible to reach into and call another namespace directly. This is the kind of things that gives a java person hives. I can technically reach into an object and manipulate private variables and methods. (The often cited quote on this philosophy difference from Larry Wall is: “Perl doesn’t have an infatuation with enforced privacy. It would prefer that you stayed out of its living room because you weren’t invited, not because it has a shotgun.”)

Now while I can, that doesn’t mean I should. There are times and places where you have to bend rules and that’s great, but I wouldn’t found my program’s structure on it. VMware on the other hand, has. Here are some of the most commonly used method calls from the API as they are in our example program in between lines 70 and 78:


The SDK has been designed for you to use these calls this way. They are not globally imported methods. Calling a function out of another namespace is something your beginning perl programmer will not understand the implications of. And the audience of  this SDK includes a lot of systems administrators, who are usually beginning programmers as part of their skill sets.

#3 Excessive use of $_ and @_

The underscore variables are both a blessing and a curse in perl. They are a very handy short-reference for one-line and similar quick-work scripting. But they are a visual headache, and because of their special status it is easy to accidentally call a method or operator that munges them in passing. Basically, it has the limitations of an unmanaged global variable even with “use strict” is on.

The block starting at line 256 is a great example of some of the visual soup this creates:

if ($_->findvalue('Guest-Id')) {
   $guestid = $_->findvalue('Guest-Id');
if ($_->findvalue('Memory')) {
   $memory = $_->findvalue('Memory');

Use named variables. Avoid $_ unless you need it.

#4 Use of foreach / bad loops

Line 254 is a great example of this. In perl “foreach” is just a verbose synonym for “for.” It’s probably picky of me, but just use “for” please.

Also, remember the soup I referenced above. Every for loop gives you the chance for a scoped and named variable. “for my $scope (@scope)” would have nicely avoided all that $_ garbage from #3.

#5 Predeclaration of subroutines

Line #20 is a predeclaration of “sub check_missing_values.” It’s been more than 15 years since that was necessary. Can we please quit cluttering code with such arcane constructs?

#6 Bang path woes

This script is using “-w” in conjunction with “use warnings.” Both do the same. The latter is the best practice. Lose the former.

It’s still up for discussion in some ways, but “/usr/bin/perl” may be better stated as “/usr/bin/env perl

#7 Passing hashes

Line 70 has us passing the complex data structure “%opts” as a hash. Technically it ends up getting passed in list context and reconstituted as a hash on the other side.

Aside from the mangling possibilities, you are copying complex data. This should be passed as a hashref.

#8 Version information

Line 18 has the script version being pushed into the Util namespace. It should be in the local script namespace, and not in Util. (The Util module should have it’s own version info.) Oh, and that variable is $VERSION and not $script_version.

If you want methods in Util to look up the version variable of the local script, it should do so. You should not populate it the other way. Better yet, it should be passed as an argument on instancing the Util module.

Dear Composer…

I’ve played a lot of band music. I’ve been playing more-or-less continuously for 25 years now and have played in a wide range of groups.

Over time I’ve come to the opinion that not everyone arranging and composing out there is treating their craft with as much care as they could. How many of us have played really poor arrangements? The kind where the music was really inhibited by the mechanics of pen and paper?

Here, without much organization are the points I’d want to write in my “Dear Composer” letter:
Continue reading Dear Composer… →

“Alerts are somewhat broken.”

“Alerts are somewhat broken.”

That’s how my more experienced coworker put it in the initial bug report. He’s a sharp guy and rarely states things in an impolitic way. So it surprised me for him to state something that direct. Turns out, it was quite true.

The customer I was currently working with wanted to receive some SNMP alerts from VMware on specific actions and events. The customer’s reasons aside, I found myself on what I thought would be a simple project: document a few alerts and how to configure them in the vSphere client, leave a nice document describing the rationale and how to do it, and leave.

It didn’t work out that way.

VMware vSphere has a bunch of alerts pre-built into them. If you’ve used VMware, you’ve seen them before. Anything that flags as a red alert or yellow warning in the status bar of your vSphere client are one of those pre-built alerts. You may have even set up a SNMP trap receiver to catch some of them for your monitoring. They are common “CPU usage high”, etc.

You can create additional alerts. And that what we were doing. We wanted to watch for were the creation, migration and deletion of VMs. Also, when clusters were added and removed and any automatic HA cluster recovery.

We set the things up and tested. And… things didn’t really work.

Events that happened on when an object was removed never happened. Events on a successfully completed cluster move did not happen. Events based upon the creation of an object happened 50% of the time or less.

So we got VMware on the phone. And started digging. It turns out that they knew of the problems.

Just as an aside here. From my perspective as a sysadmin, for your product to have problems is ok. I’d like it to be perfect, of course, but I also know that reality is rarely perfect.

On the other hand, I’d like to know when I am opening Pandora’s hard drive. Tell me when your product has sharp edges and I’ll avoid getting cut. No where have I seen a KB article or release not acknowledgement from VMware that their monitoring has issues. But if you raise enough racket in support, you find out they know.

And just for perspective this is SNMP-FRIKKING-MONITORING. This is what people are going to use and expect to work as a part of your product. This not a kid-of-the-CEO-in-the-corner project. VMware is positioned dead center in enterprise land. I’ve installed a lot of crap software, but as long as I could keep an eye on it and maybe automate workarounds, I felt it was ok to go live.


It is broken. And if you end up on this post at the end of a tortured google search, here are the SR’s you need to ask VMware about:

  • SR#1502635871 – PR#530996 -Removal events (host, objects, clusters, etc) do not alarm, as they have no object to bind to within the event system.

With a little poking in the logs, we were able to figure out what is going on here. When an alert is triggered, internally vSphere attaches the information from that alert to the given system that triggered it. It makes logical sense in design. If you have a “CPU 100% alert” and a system trigger’s it you want to see the red-flag and alert attached to that system in the GUI. You want to see “System A had a CPU 100% alert.”

So what happens when you alert on a delete action? Well. The alert is triggered as you delete the object and vSphere attempts to then attach the alert info the the object that is now gone. If you up the log levels on you vCenter server, you can see the object error get thrown. Nothing gets displayed in the GUI. I guess it is a bit of a philosophy in design difference. I would rather have a failing system toss to the end user an “unknown error” instead of losing an alert silently, but I guess hindsight is always 20/20.

  • SR#1502635871 – PR#538702 – Creation alarms can be intermittent due to object-creation race conditions.

These were the alerts that were 50% or less on triggering. But it’s the same cause as above. The creation of the object or resource is obviously asynchronous. (You wouldn’t want the GUI to freeze until something is done being created. So it starts the process and returns.) So I am guessing this is a classic race condition. If the resource is actually created before the alert finishes executing, it attaches and sends. If the object is not there to be attached to, it bombs silently into the logs as above.

  • SR#1502635871 – PR#539840 – The successful accomplishment of a VM failover within a cluster does not trigger and alarm.

Nothing in the logs here. This just plain doesn’t fire. Considering that the clusters are handled by the ESX hosts within the cluster and not directed by the vCenter server, I’m assuming they just don’t bother to surface the event up to the vCenter server on success.

The notes on the listed bugs above are a bit interesting. One of the comments that came back was these features are slated to be removed.

As a final note, if you end up handling SNMP events from VMware, you might notice something interesting. For the more triggers you have on an event, the message of the SNMP trap is duplicated in text seperated by ” OR “. That can be found here:

  • SR#1508061687 – – Content of SNMP messages is duplicated when multiple vCenter alarm triggers are present.

Mental music!

If I could crack open my head, this is what this day has sounded like:

Bevete piu’ Latte

(It’s from the Italian equivalent of our “Drink Milk.”)

There. Now it can be stuck in your skull for a while.

New Toy!

LeBlanc Contralto and Contrabass clarinet.

Can haz new toy! Isn’t it beautiful? Its baby brother is next to it.

They are such a wonderful blend of 18th, 19th and 20th century technologies. So many shiny gears and levers.

Put these suckers in my car and it doubles in value. I hope to have it overhauled in time to play on the fall concerts.

“I’m in the flat part that gets all the sun with the mountains around it…”

So… Phoenix arizona gets hot. It’s a middling mix between San Jose and burning man. At least weather-wise.

I hear that during a heat wave out here, when it gets above 120 F, they have to shut down the airport because the planes don’t generate enough lift to take off, and the tires melt on the tarmac as well.

That being said I had some good, proper, BBQ out here.

This was my first solo gig. Standing up in front of a customer lecturing on a product and helping tune it for their situation and needs. I think it has worked out very well. I’ve been here a week and we’re on break right now on my last day.

The hotel I stayed at reminded me a lot of the old Anthrocon Hotel. Being one of those “Starwood” branded hotels it had a lot of the style and finish of the Adams Mark. I was expecting to run into a fursuit sooner or later. But jsut businessmen.

Oh well. Back on a plane and home for the weekend. And then they’ll probably ship me out again from the look of it. Oof.