Caught in a (vm)Trap

I was reminded by a recent posting by Joachim Tuchel (VMTRAP with VA Smalltalk due to strange bytecode differences (or whatever)), that I needed to document my own recent experience with vmtraps.

I was working with another developer at TAF debugging a VA Smalltalk 8.5.2 image in AIX when his development image died, and couldn’t be re-opened. Moments later, other developers using the same AIX machine reported that they couldn’t open development images either. Packaged images were fine, as were VAST 8.0 images. Lacking a better solution at the time, I had our system administrator bounce the box, and all was well.

I was later reminded by the fine folks at Instantiations support that I should have checked for rogue es processes, and made sure that the /tmp/es directory was clear of all files once every possible Smalltalk process was gone. The same reminder would hold true for Linux and Solaris users as well.

Just something to keep in mind when working with Smalltalk in a UNIX environment…

Where Was I?

After many months, once more in a position to think about VA Smalltalk upgrades and the automation of packaging again. When 8.5 came out, it broke the tools I developed during the upgrade process from 7.5 to 8.0.2. Fortunately, subsequent releases have fixed things, so I’m at a point where I can complete the upgrade to 8.5.2. I wrote about many of these issues in an earlier blog entry.

As is my usual custom, I’ve started by modifying my config maps to specify either 8.0.2 or 8.5.2 maps as prerequisites. The development environments had few problems, but packaging was a challenge. Instantiations has added some interesting new time zone support to the base image, but it requires one to include an additional application, EsTimeZone, as part of the runtime package. If one is creating a new package from scratch, the default rules will apparently pick up this application automatically. The rest of us are not so lucky – we have to modify our packaging instructions to include this new package. This new requirement isn’t currently mentioned in the upgrade notes.

In my case, because I’m building 8.0.2 and 8.5.2 images from the same code base, I’ve been forced to move the packaging rules class to a subapplication. A configuration expression based on the system version loads the appropriate subapplication. This is where I found one more little “gotcha”.

In 8.0.2 and earlier versions, the preferred mechanism for determining the Smalltalk version was to send #vaVersion to System. This has been deprecated in favour of #imageVersion, which answers the same thing. If you’re like me, and trying to maintain a couple of versions of Smalltalk, you’ll probably find it easier to continue to use #vaVersion for now, and switch to #imageVersion once you’ve retired anything prior to 8.5.

Other than these few minor annoyances, it looks like there haven’t been any significant changes introduced, unless you were an early adopter of the log4s framework. If you were, you’ll want to look things over a little more closely.

Fit-ting In

Here at TAF, we currently run a mix of generic Linux PCs and thin client netstations as X terminals for our AIX production box. The netstations have long been unsupported, but we’re also interested in something more robust and less power-hungry than the PCs we’re currently running. One of my tasks recently was evaluating a Fit-PC2 for use as a replacement.

Out of the box, the Fit comes with Linux Mint installed. For those not familiar with it, Linux Mint is based on Ubuntu. In this case, we’re talking Linux Mint 9, “Isadora”, which corresponds to Ubuntu’s 10.04, or “Lucid Lynx” release. This isn’t a bad thing as it’s a long-term support release, but Ubuntu and friends are problematic for the use we have intended for these machines.

In order to connect to AIX, we rely on XDMCP. We also rely on xfs to provide certain fonts for use by the programs running in AIX. The Ubuntu distributions have removed this support by default, and made it an extreme challenge to put it back in. I spent a day or two fighting the Fit to re-enable this support before giving up.

My next step was to take the brain-dead approach of installing Red Hat Linux using the same install CD and kickstart script we’ve used for our other Linux installations. This actually worked, although disk and video performance weren’t outstanding.

The Fit is based on Intel’s Poulsbo chipset, which isn’t natively supported in our version of Red Hat. Some research in the Fit-PC2 forums yielded a kernel module that provided a working driver, but given the somewhat informal source and the fact that it was for a different distribution, I was reluctant to recommend it for production. However, I did try it on our little box, and was pleasantly surprised. Everything seems to be working as expected, and we will be throwing the box into a production test sometime in the near future.

As a longer term solution, we need to upgrade to a more modern version of Linux along with a configuration tool such as Puppet. RHEL 6 seems to be headed in the same direction as Ubuntu in terms of XDMCP support, so we have some work in store for us.

No Pain, No Gain?

After a bit of poking around, I’ve started the job of ramping up the code base at TAF to the new release of VA Smalltalk, recently announced at ESUG 2011. While there’s lots to admire about this release, a couple of things have almost stopped my upgrade efforts in their tracks.

I was actually looking forward to the deprecation support, until it bit me in the posterior. In their “wisdom”, Instantiations decided to deprecate EmSystemConfiguration>>#vaVersion, which has long been the preferred way of identifying the version of VA Smalltalk, in favour of EmSystemConfiguration>>#imageVersion. This, in itself, would have been all right, except that the deprecation implementation is broken, such that an exception is raised regardless of any tolerance settings. At the very least, this makes the config map browser impossible to use.

The other problem I’ve encountered is more subtle. You might recall from an earlier series of posts that I’ve automated the building of images based on a virgin distribution image, some code, command line parameters and Jenkins. Sadly, there’s been some change in 8.5 that breaks this. Something goes seriously wrong during processing of the abt.cnf file, leaving one with a walkback. The workaround is to start a virgin image, let it cache to the correct repository, reset the image owner, and then save and exit the image. This new image should be used as the basis of future automated builds. It also allows for the opportunity to load a fix for the broken deprecation implementation.

I dislike having to take this approach – it leads back to the “magic build image” situation that I’ve been trying to avoid all the way along, but, so far, I’m defeated…

An Alarming Requirement

A little while back, one of my colleagues at TAF came to me with an interesting challenge. He had been assigned the task of making a production application raise a visual and audible alarm in response to invalid actions on the part of the user. It doesn’t seem like that hard a problem until one realizes that our applications are actually running in XWindows sessions on an AIX server. This means that the running application thinks that its environment is the AIX server, and it knows nothing about the machine actually displaying the user interface.

In the past, this sort of requirement was handled by using a Programmable Logic Controller (PLC), and running conduits and wires from the PLC to the location where the output was wanted. This gets the job done, but it’s expensive to install and move, and requires a lot of lead time if a change in location is needed. On top of that, one also has to involved the services of a PLC programmer, and it may even require the acquisition of an additional PLC if a free port is unavailable on one of the existing alarm-capable PLCs. We needed to come up with a cheaper, more flexible solution.

At first, it was a bit of a head scratcher, but, as it turns out, an application running under XWindows can get one vital piece of information about its environment: the IP address of the machine actually providing the display. In VA Smalltalk, the command “‘DISPLAY’ abtScanEnv” will give you a response of the form ‘xxx.xxx.xxx.xxx:0.0’ where anything up to the colon (:) is a host name or IP address, and anything after is the XWindows display number. In our case, we’re only concerned with the host name or IP address. If the XWindows display is actually running on the same machine, the host name will be blank, and you’ll get back something like “:0.0”.

Knowing how to get to the physical computer, we can now think about what to do there. While I have complete control over how the client machines are loaded, it’s generally preferable to keep the installed software to a minimum. Everything I needed to do could be done from a UNIX shell prompt, so using some shell scripts seemed the best answer. Modern Linux implementations make it easy to set up TCP daemons using xinetd, so I set up a listener on a TCP port that invoked a simple script.

My Smalltalk program opens the port on the client machine using Totally Objects SocketSet, writes a value to the port, and closes it again. On the client machine, the script reads a line of input from STDIN, and uses that input to invoke an appropriate alarm script. You could do the same thing with VA Smalltalk’s socket support, but if you value your time at all you’ll quickly come to appreciate the benefit of Totally Objects’ product.

This has been working quite well, but a new curve was thrown at me recently. Instead of just firing off an audible alarm, the users wanted the computer to send a voice instruction. Since the number of instructions was fairly small, it wouldn’t have been impractical to record sound files and play the correct one based on an alarm number, but it wouldn’t scale well and keeping clients up-to-date would start to become a challenge.

Instead, a little research turned up the fact that our preferred Linux distribution, Red Hat, includes a speech synthesis system, Festival, with text-to-speech capabilities. Even better, Festival can be run as a daemon that listens on a TCP port. A little bit of shell programming later, and Festival was wrapped as a Linux service. Another bit of Smalltalk code using Totally Objects SocketSet was used to pass text to that service, and we now have generic text-to-speech capability in our application.

Certainly Smalltalk could have been used to implement most of the daemon side of these alarms as well, but some scripting would still have been required. Using the approach we did allows us to keep the installation package light and minimize the skill set required to maintain it in the future. Moreover, testing, troubleshooting and modifications can be done easily from a command prompt the client machine which has a lot of value when you’re trying to fix a problem in a factory a long way from your development environment!

You Can Never Go Home, or Can You?

Over the past several weeks, I’ve been picking up the pieces of a project that I thought I’d finished when I left the TAF back in 2005. However, it seems that my past has returned to haunt me, and I’m once again digging into a server that I’d implemented using VA Smalltalk’s Web Connect and Web Services frameworks.

It’s an eerie feeling returning to a project like this after six years, sort of like returning years later to the house you grew up in. A few changes may have been made here and there, but you can find your way around in the dark. In the case of my project, a couple of people have hacked on it during the intervening years, but the code was largely intact.

One of the biggest initial hurdles I had to overcome was the fact that someone had decided that a slightly different version of the code was needed for use in one of the plants at TAF. The server had been running happily unchanged for over a year. Normally, I would have continued to ignore for a while longer, but it was the last part of my Smalltalk upgrade project, so I had to be able to rebuild the server. This proved to be a challenge.

One of my hard and fast rules is that one should be able to load one single config map into a virgin image to create a specific version of a packaged image.In the case of this server, the specialized code was gathered into its own config map, but the config map itself had no required maps defined. As a guide for building the runtime, the map was completely useless. The map contained applications that were in more general config maps that built other images, but different versions with changes for which there was no documentation. Some of the code changes appeared to be gratuitous, and in other cases, there were methods that referred to methods that couldn’t be found anywhere in the code repository.

I eventually managed to shoe-horn the contents of this map into one of the existing complete maps to complete my migration, but we’ll never know what code was actually running prior to the migration. All we know is that things were working before the migration, and they continued to work after the migration. With the migration complete, I was free to concentrate on the more interesting problem of figuring out why some things that should have been working weren’t anymore, and adding some interesting new functionality.

Browsing Processes

James Robertson’s Smalltalk 4 You 87 looks at process debugging in VA Smalltalk. This reminded me that there’s a great tool wandering the interwebs that makes the task even easier. Back in the dim and hazy past, Jeffrey Odell created the VA Process Browser, a companion to the SUnit Browser that’s found in VA Smalltalk today. It looks like this:

As you can see, it gives you a nice list of all processes currently in the image, along with their current status. The display updates on a periodic basis, or you can manually refresh it. All the things you could reasonably want to do with a process are easily accessible, and the tool itself is launched from the Tools menu of the Transcript window.

Unfortunately, Jeffrey appears to have moved on to other endeavors, and the VA Process Browser code has fallen by the wayside. If you’re out there somewhere, Jeffrey, it would be great if you’d make the source available on VAStGoodies.com, or, if you send the necessary permission, I’d be happy to do it on your behalf. In the meantime, interested parties will need look around or ask the right person to obtain a copy.



Follow

Get every new post delivered to your Inbox.