Using a Xen virtual appliance to manage your VMware servers

One of the great things about virtualization is the amount of money you can save companies, not just in hardware costs, but also the on going costs of keeping machines in a data centre and a rack, and feeding them with power and air conditioning.

Well, I built several ESXi servers for the current bank balance filling company, and they needed an easy and affordable way to manage them on the cheap. Now, I have always thought that VMware workstation is very affordable, so I built a Vmware virtual appliance to look after the servers with Vmware vSphere client. This however proved too expensive for them and they wanted something at zero cost to the company.

It is obvious that the management virtual appliance has to be on a different virtual infrastructure to that which it is supporting, so I built another virtual appliance , this time on our Xen server. As the virtual appliance has to be built with windows XP as it's base and then security hardened, the Xen server has to have the Intel processor enhancements turned on in the BIOS, otherwise it just won't fly.

The Xen server is based on RedHat Enterprise Linux 5.4, which comes with a lot of enhancements to the virtual machine management program - virt-manager, and it is very straight forward now to configure new virtual machines.

All I had to do to allow remote management of this appliance was to turn on the terminal services process within XP, and now people can use the "remote desktop connection" program to access this VM and control the ESXi servers using vSphere client.

Xen offers great performance, and this solution works really well, and didn't cost a penny.

Playing with Google Chome OS in a virtual machine

After watching the Google announcement on Thursday, I decided to build a virtual machine and compile the operating system from source. I used an Ubuntu 9.10 virtual machine , as that is what they recommend, and then followed the build instructions from the developers web site. Eventually after an hour or so of compiling , it spits out an image you can put on a USB stick.

I then booted another virtual machine from this USB stick and built the following image. You log in with your Google account and away you go. I have to say considering this is an alpha release, it works and is not that slow to have a look at. I'm really enjoying the chess game, it really shows what can be done on this platform, and anyone who thinks this OS is just going to stay on netbooks is deluding themsleves.

I would have to say, that unless you like playing with the source code, which I do, then there are lots of people publishing pre-built virtual images for you to play with, and that would be a lot easier way just to have a look. If you do want to do the compiling yourself, then here is the link - building Chrome OS from source

Here's the screen shot:

Chrome OS built from Source

Here is the demo.

If you want real virtual speed with your Jumpbox appliances use Xen not ESXi

As you will have seen on my blog recently we had a situation that meant I had to move one of our Jumpbox virtual appliances from a failed Xen server to a new ESXi server. Now these servers are exactly the same specification, HP DL380 G5 with 32GB RAM and 2 x Dual Core Intel Xeon CPU's , so not exactly supercomputers, but more than enougth to do what we want with them.

Well here is the thing, Jumpbox say that VMware are one of their main supported platforms and , Tier 1 partners I think they call it, and open source Xen is Tier 3. So how come then the virtual appliance runs significantly faster on Redhat Enterprise than it does on Vmware ESXi? When I say significantly, I'm not joking. The virtual appliance is the moinmoin wiki. On the ESXi platform trying to edit a post is painfully slow, 30 seconds to open the editor. On Xen it is 1-2 seconds at most, so I don't need a stop watch to see something is wrong.

I have gone through the settings for the Vmware server, something I don't have to do with Xen, and given it garanteed resources and CPU power, no difference. I could, if I fancied wasting a small part of my life, investigate the Advanced settings for the ESXi server, but again why would I when with Xen it just flys?

I don't know the answer thats for sure, and especially when you know that the Jumpbox appliance is running with the QEMU hardware emulator on Xen, as it's not para-virtualized! Emulators as I'm sure you know are not supposed to be that fast.

I think I'm going to chat to Jumpbox about this, as it really does seem odd. The only thing that may have affected the performacne is the fact that I had to migrate the appliance to ESX format from Server format, but surely once this process is completed the virtual image should be good to go.

Google Chrome OS is alive and open to the community to develop

I sat and watched the webinar this evening that launched Google Chrome OS into the wild at Google's headquarters in California.The news about their new web centric operating system is very good indeed, and they truly have thought the concept through.

Now, I'm sure the open source community will take this rough diamond and turn it into a beautiful gem, but what has already been shown is interesting enough. The system is building upon the great work of many very successful open source projects like the Linux kernel, Ubuntu, Moblin, Webkit and of course Chromium. Once the system is booted and running, you wouldn't notice a difference between running the Chromium browser in full screen mode, which I'm doing now to type this. It is this envelope use of the browser to drive the whole experience which is new and cutting edge.

The new system will be free to download and install on any system, but is being aimed at a set of reference hardware that will have 100% compatibility out of the box , and is going to be offered by several major hardware manufacturers at launch.
This bringing together of the hardware and software offers major advantages, as Apple have found, so that the whole experience just works. They have also written there own BIOS, which again will work well with the other components.

Google have also thought long and hard about security which is designed into the system from the ground floor, with multiple layers of sandbox, to protect running applications, and the fact the OS checks for updates at boot. The speed to get to the browser has also been addressed and is a matter of seconds, so the whole suspend/hibernate cludge disappears in a cloud of blistering fast bits.

The number of applications with the new native client - think of Google gears on steroids - will be virtually infinite, and the ability of Google to create and grow a web store has already been proven with the fly away success of Android. If I were a web based developer, I know where I would be investing my energies over the next twelve months, as this could be a gold mine.

I have loved the Chromium browser since it arrived, warts and all, and I have no doubt that this new OS will find a place on my netbook. I agree with the usage profile discussed at the launch, and it is unlikely to replace Ubuntu on my laptop anytime soon, but my netbook is crying out for this.

To use a phrase that this launch fits well, the operating system has just evolved.

Mozilla Weave sync application heads towards version 1.0 release

After updating this extension for Firefox this morning I noticed the following information from the Weave team

"Latest Weave Update
Weave 1.0 beta 1 was released on November 16th, 2009

* Improved Firefox preference integration with sync direction choice on first-connect
* Fixed issue with connecting on Fennec 1.0b5
* Added an automatic on-demand sync if many changes are detected
* More comprehensive history sync (initially and continuously)
* Many added bugfixes

Release Notes

1. Upgrade all your clients to the latest version
If you are using Weave Sync on multiple computers, you will need to update all of them to the latest version. Weave Sync requires that you use the same version of the addon on all your computers that you want to synchronize data to/from.
2. Syncing tabs across multiple devices requires configuration
If you are using Weave Sync on only one computer, it will not sync your tabs. After you install Weave Sync on any additional computers, you can enable tabs sync. To do so, on your original computers, open Weave Preferences and ensure the "Weave will" option is set to Sync Everything. Force a sync by choosing Sync Now on all of your computers and you should now see tabs syncing.
3. Server APIs
For this version of Sync, Mozilla no longer supports the old database and server API versions (i.e. before 1.0). You will be unable to use any version of the Sync extension that is before 1.0 beta 1 with the Mozilla servers.
4. Hosting Your Own
If you are hosting your own Weave server, you’ll need to add an apache alias line for the 1.0 URL – your old server will still handle the same API "

It is worth noting that with this release you will need to update all your clients as they have obviously changed some components in the backend.

This is a great extension for Firefox, and I hope that the Chromium bookmark sync version will eventually offer the range of features that this does.

Extensions coming soon to Chromium on Linux so they are now telling us

The screen shot below says it all.

Extensions library

The launch of this may be timed with the big announcements coming tomorrow from the Googleplex or maybe it is completely unconnected, but all will be revealed tomorrow no doubt.

Google community looks ready to launch a new extensions site

With the latest builds of Chromium on Linux there is a new link that has appeared which is similar to the curtain for the themes that are now available.

At the moment it takes you to a Google search screen, but it is obviously limbering up to offer a whole new site full of Chrome extensions. To be fair, Firefox is not going to have to worry too much for the immediate future as they already have thousands of excellent extensions, and it will literally take years to get to that level.

Here is the link.

Bookmark syncing arrives for Google Chromium on Linux

I have long talked about the advantages of Mozilla Weave and how you can securely keep you browsing information in a cloud based repository so that it can move with you where ever you decide to do your browsing. This is extremely useful and saves having lots of different bookmarks on different computers.

Well the ability has now arrived for Chromium, and it is built into the browser rather than an add-on like Weave. To enable it you need to call the browser with the

--enable-sync

setting, and it will appear in the wrench menu, see below.

Chromium sync options

The Chromium version works in a completely different manner to Weave in that once you have authenticated your Google account information your bookmarks are stored in your Google docs area as a folder. This does have the advantage that you can easily share your bookmarks with friends and family if you wanted.

This facility doesn't offer the full range of choices that the Mozilla Weave does, but it is the first iteration and I'm sure this will be developed further.

You can check the status of your synchronisation in the options menu, again see below.

Firefox 3.6 adds performance and some nice new touches

As the battle for functionality rages between Mozilla Firefox and Google Chrome and Opera on the Linux desktop, the features list and performance improvements just keep coming. The latest version of the javascript engine in FF 3.6 offers a supposedly 15% increase, and it does indeed seem faster on my Dell XPS 1330 with Ubuntu 9.10 64bit version.

One of the nice new touches you can have with the latest releases of the 3.6 version of Firefox is the tab preview window, see the picture below. Now at the moment there is not an easy way to turn this on, but just follow these simple steps.

1) open a new tab with CTRL-T and type in the address bar about:config

2) Press the "I'll be careful" button

3) Type this into the selection bar - "browser.ctrlTab.previews"

4) Double click on the item when it appears to turn it from false to true - it will enbolden.

5) Re-start Firefox

6) Press CTRL-TAB to see your tab previews.

This is similar to the nice feature in Chromium that you see when you open a new tab, Opera also offers a tab preview by dragging the tab window.

Give the Beta a try and see what you think.


Firefox 3.6 tab preview enabled

Upgrading VMware tools in your Jumpbox virtual appliance

If you migrate a Jumpbox virtual appliance between virtual hosts, for instance, form a VMware Server or VMware workstation to an ESXi server, the already installed configs and installed drivers maybe out of date or the wrong version, and the management screens will complain and let you know about it. This is very easily cured.

Firstly , make sure you have ssh enabled on your virtual machine, and log in as root or admin. Once logged in, go back to your management application, which for ESXi will be the vSphere Client, and select the virtual machine and select install vmware tools from the right button click on that virtual appliance.

This will have effectively mount an iso with the files on, which you must now mount from your ssh terminal window using a command like this:

mount /dev/cdrom /media/cdrom - (you may need to create that directory)

They move into that directory and cp the file VMwareTools-4......(whatever).tar.gz to say /tmp

Go into /tmp and untar the above file, and then move into the directory it creates.

Run the following command and hit enter when asked, after reading the output, but for a standard Jumpbox, just hitting enter will do:

./vmware-install.pl

At the end of the process you can either reboot the virtual machine, or type in the following command:

/etc/init.d/networking stop; rmmod pcnet32; rmmod vmxnet; modprobe vmxnet; /etc/init.d/networking start

Make sure you have the semi colons and type it all on one line.

Voila, you now have the latest vmware tools, which means you are running the optimum networking drivers which is important for performance

Moving a Jumpbox virtual appliance from Xen to VMware ESX

We had a server failure here yesterday , which was a standalone Redhat Xen virtual server with a few non critical applications on, thus the single server. While we were waiting for HP to get a new motherboard delivered I decided we needed to get our Wiki back and running quickly, and the only available virtual server was one running VMware ESXi.

When you download a Jumpbox appliance it comes ready to run on several types of virtual environment, but VMware ESX is not one of them. It happily supports VMware workstation or server, but the filesystem on ESX is different.

There are two ways to convert the virtual image, the easy way or the hard way, both work.

The easy way is to download the VMware converter utility which basically takes the uncompressed virtual appliance you downloaded, and automagically uploads it to your ESX host , if you have it on line. You can just convert the appliance and upload it yourself later if you prefer.

Job done. I could then get on with setting the IP address of the virtual machine back to it's original setting and restoring from backup it's complete configuration and data. Took about 20 minutes, but you would need to factor in the size of your appliance, the speed of your network and the size of the backup. That is one of the great things about Jumpbox virtual machines, their backup and restores are a nailed on perfect solution and have worked here everytime. We happen to use an NFS mount on our NAS server, but you can back up to Amazon EC2 if you were web connected.

The second way , is to manually upload the virtual appliance files to the ESX datastore, and then run the vmfstools from an ssh shell. Setting up the shell on ESXi is a job in itself, go here or here for more info.

There is a detailed article on doing this here.

I think you will agree that using the converter tool is just a little easier ;-).

Another very useful feature of the Ubuntu live CD

I had to look at an Ubuntu installation the other day that had a problem after it had been upgraded. This problem turned out to be that the friend had previously installed an early version of the Grub boot loader and had not regressed this before the upgrade , an easy mistake to make.

The cure for this is well documented on the web and the solution is to boot your PC with an Ubuntu live CD and mount the harddisk in a chroot jail so we can then play with it and re-install the correct version of Grub. This procedure was taken from the official Ubuntu wiki.

Reinstalling GRUB 2 from LiveCD
If you cannot boot from GRUB 2 and need to reinstall it:

* Boot to the 9.10 Karmic LiveCD Desktop.
* Open a terminal - Applications, Accessories, Terminal.
* Determine your normal system partition - `sudo fdisk -l` (That is a lowercase L)
* If you aren't sure, run `df -Th`. Look for the correct disk size and ext3 or ext4 format.
* Mount your normal system partition:
Code:

sudo mount /dev/sdXX /mnt

o Note: substitue the correct partition: sda1, sdb5, etc.
o Note: GRUB 2 counts the first drive as "0", but the first partition as "1"
* Only if you have a separate boot partition:
o

sudo mount /dev/sdYY /mnt/boot

with sdYY being your /boot partition designation.
* Note: If you have any other system partitions such as "/usr" these should also be mounted in a similar manner.
* Mount devices:

sudo mount --bind /dev/ /mnt/dev

* Chroot into your normal system device:

sudo chroot /mnt

* Reinstall GRUB 2:

sudo grub-install /dev/sdX

*
o Note: Substitute the correct device - sda, sdb, etc. Do ''not'' specify a partition number.
* Verify the install:

sudo grub-install --recheck /dev/sdX

o Note: Substitute the correct device - sda, sdb, etc. Do ''not'' specify a partition number.
* Exit chroot: CTRL-D
* Unmount devices:

sudo umount /mnt/dev

* If you mounted a separate /boot partition:

sudo umount /mnt/boot

* Unmount last device:

sudo umount /mnt

* Reboot.


This procedure had his machine up and running perfectly again in five minutes, but just shows how flexible this live CD is and there are no real limitations to what you can get up to once you have this up and running.

Is ranting about Microsoft Windows on a par with being a religious zealot

Having studied evolutionary biology for most of my life, an interest I picked up in school, I have read a lot about many facets of the Theory, and one side junction from this is the concept of Memes. This has been described as a social/cultural equivalent to genetics and has many postive qualities that has helped the progress of Homo Sapians to evolve to our current state.

One simple example of this is the ability of parents to pass on to their children the concept that swimming with crocodiles is not a good idea for the families genetic advancement, but I think you get the picture. This has also been used as an example for the worldwide adoption of many types of religions that get handed down through the ages from parents to children with not an ounce of evidence but just firm beliefs.

This has led me on to thinking about some peoples rabid support for Microsoft Windows when the facts presented to everyone run completely contrary to the evidence. I would argue that it is Memes at play here , and I'm sure that Microsoft use this to their advantage. This post isn't going to bash Windows or put forward Linux per se, I use both and they both have there place. It just facinates me why people can firstly be so "religious" about an operating system - removing Microsofts huge PR campaigns for an instance. I have used Apples MAC OSX and it is a good system, it has some issues, but for a home user it has many advantages. It's one big draw back for me is the cost of the hardware, but thats just personal. I have used BSD, Solaris etc, and they are also good and have there place in the operating system eco system. I use Microsoft XP in a virtual machine for certain non-other-os applications, and it works fine in that role.

I choose to use Linux for many reasons, none of them fanatically idealogical. The first is that it is intellectually interesting to working within a software development environment where you can be actively involved, which I am. I talk regularly to many developers of many systems about issues and bugs, try doing that with Apple or Microsoft. The software that the various communities produce is of a very high standard and can be used in many interesting and diverse scenarios, with everything from running my desktop to type this blog post, to a mutli-million pound cloud infrastructure, all with the same basic building blocks. I also enjoy the freedom that the developers have to create many interesting and complete solutions to web based problems, just look at the most famous open source project of them all Firefox. They can just grab the components they want without worrying about costs or licenses, and I know personally that is a big plus for me.

I feel that people who enjoy the variety that the modern software world has to offer have open minds and are happy to explore new ideas and paradigms, and that the closed minds just follow the hurd into the same old ways of doing things, the Microsoft way. This is my experience and your mileage may vary, but how many MCSE's do you know would install an Apache Tomcat server to just get the job done, instead of trying to stear the entire project to use .NET?

It appears to boil down to one simple fact, that Microsoft got there first and have placed their Memes firmly in place, and like religion , it is going to take one hell of a lot of shifting. I feel with things like Andoid and Maemo people will see on their portable devices how good alterantives can be and this will start to break the mold. I also see Googles OS having a major impact to affect change in peoples minds, but it will not be quick or over night.

One thing is perfectly clear though, Redmond are not going to give in without a fight, and they know how to be extremely dirty when they have to. Anyone see the parallels here to jealous, religious zealots? I certainly do.