Using puppet with Virtualbox to speed up client additions install

One of the first things you need to do when you create a new VM is install the client additions code, which offers enhanced networking,screen sizing, cut & paste and general host/client integration. With Centos 6.3 Live CD installs , several of the needed rpm packages that allow the install are missing.

I have a puppet master installed and configured - which is beyond the scope of this post - but if you need help doing that, then just buy or download the Pro Puppet book, which covers this topic very clearly for any operating system.

Pro Puppet book

Make sure every machine can resolve all DNS entries properly , either with a DNS server or by hosts files - if you must. Every machine needs to resolve puppet.mynetwork.com and puppet.

You also need to install ruby , ruby-libs ,ruby-shadow , puppet and facter or you won't be able to connect to the puppet master in the first place. You can do this as part of your post-install kickstart or manually. You will also need to add the Fedora mainatained EPEL repo to get the latest versions of these.

The important elements that you need to install are make, gcc and kernel-devel and I also always add vim.

You need to define each of these in the modules sections under /etc/puppet/modules  on the puppet master with their own directory structure of:

/etc/puppet/modules/make/files
/etc/puppet/modules/make/manifests
/etc/puppet/modules/make/templates

Within the manifests directory you need to add an init.pp, and for vim it will look like this:


class vim {
package { vim-enhanced:
ensure => present,
}
}

You can obviously do a lot more in these files, but this is just to show what is needed for this task.

You also need to make sure you include these modules in the node.pp file in the /etc/puppet/manifests directory. Like this:


node /^testmac\d+\.mynetwork\.com/ {
        include vim
        include make
        include kernel-devel
        include gcc
}

The guest server in my environment follows this naming scheme and the regular expressions allow you to have as many as you like.

To connect your new VM  to the puppetmaster you can run the following command and make sure that  the certificate is allowed on the server:

puppet agent --server=puppet.mynetwork.com --no-daemonize --verbose

Once the client server has the cert's installed correctly, then run this command again to auto install all the rpms needed for you to install the client additions. The client additions can be installed from the GUI or the command line, which ever you prefer.

You can now also manage all your VM's with puppet , and make your virtual dev/test environment a lot easier to manage.

Putting VNC traffic through an SSH tunnel

While setting up ssh tunnels has become a useful way to get work done securely on the command line, there are occasions when a GUI is required. Enter the really useful VNC protocol for being able to remotely access a Linux machines X windows system. So how is this done, well I think a little diagram will help to explain the idea, and then some steps to get it done.
 



Well, how do we get this done. Read my previous article on how to set up tunnels using SSH, and once you have that up and running we can set up the VNC servers. Using Centos 6.2 as the server to connect to internally, running Gnome, install Tigervnc server and edit the /etc/sysconfig/vncservers and set your X session size, display port numbers and we wish to tell it to listen only to localhost. This is for security, we don't want it to respond to VNC on the actual IP address of the server.

Example:
VNCSERVERS="1:nhaddock"
VNCSERVERARGS[1]="-geometry 1024x768 -localhost"

Start the vncserver , which will now be listening on port 5901 on localhost. Next off to the remote machine, where we need to set up the SSH config file so that it tunnels all VNC traffic through the SSH tunnel Go into .ssh/config and add the following for the remote user/machine

Host ext.server*
#HostName ext.server
User joeblogs
UserKnownHostsFile ~/.ssh/known_hosts.d/companyx
LocalForward 5901 localhost:5901
ProxyCommand ssh -A server.in-the-cloud exec nc localhost 10022


The key things to take away from this config, is that you need to add an entry in your /etc/hosts file for the ext.server IP address. We are going to forward all traffic for VNC - port 5901 into the tunnel once it is established using Localforward, and the proxy command uses netcat (nc) to push all the traffic down the reverse tunnel port it will find from your previous set-up with the reverse tunnel. So now just ssh to ext.server, and once you have made your connections, use a VNC viewer to connect to localhost on port 5901, and voila you now have an X session on the internal server.

Here are some pictures of the outcome.

eSATA , Thunderbolt or USB3, that is the question?

As with all things technological , when a standard comes to the end of it's life there will always be more than one way to move forward.

So it is with the demise of USB2 and Firewire, which have basically come to the end of their usefulness, when you look at the size of the external disks we now have , and the size of the data that we wish to move around.

There are three contenders that I can currently see to replace these, and they are USB3, eSATA and Thunderbolt. Now I think that Thunderbolt is a great technology and offers the best performance without a doubt, but currently it only seems to be getting traction in the Apple world, where even here external disk drives are thin on the ground. To compound this scarcity of choices is the fact I live in a multi-machine world, where I have to move data between many machines, which include mainly Linux and even the odd Windows VM.

The PC manufacturers do not seem interesting in this new interface yet, I hope this changes, but I have to deal with the here and now and not what I would like to happen.

I initially started using eSATA as my laptop supported it, and it was reasonably fast and I could run VM's directly from this drive - using a hybrid disks - but it has a big hole in it's usefulness, not every machine has one of these ports, and it is not compatible with anything else. USB3 on the other hand is completely backwardly compatible, and this is where it really wins, as I can use USB3 where I have it , and it can switch back to USB2 when it is not. This has proved more than useful on many occasions while moving files around.I have also noticed that more and more motherboard manufacturers are supporting it, and once it becomes a standard on laptops, then to my mind it is game over. I now see eSATA as more of a bridging technology that became popular due to the absence of the the new USB3 standard.

So with this in mind I have just bought a USB3 PCI-e adapter for my older Desktop server, and I will now compare this performance to the ESATA I have in the laptop and see how I get on. All the benchmark data I have read lead me to believe it will be significantly faster, as it's speed is closer to the 6GB SATA III than the 1.5GB SATA speeds I currently have in the laptop. I will report back.

I always feel on these technology change overs that you need to be careful, so that you make sure you cover your bases and examine the alternatives, but need to make sure your not left holding the new Betamax technology.Anyone still own an HD player when you now wish you had a Blueray player?

There is one other part of the jigsaw that I will right a separate post about, and that is a the filesystem to use to make the external disk truly portable between Mac, Linux and Windows.

The power and felixibility of OpenSSH tunneling in the cloud

If you didn't realise it, the secure shell that is used every day, all over the internet is written and managed by the OpenBSD project team, and we all owe them a huge debt of thanks for their efforts. It lets us communicate securely with our servers and other devices.

Well, one of the great features of ssh , is the ability to set up tunnels , that you can use to connect traffic across , and indeed reverse tunnels which can be very useful for connecting to remote servers.

As this tool has become so ubiquitous, there are other tools that have been built around it, and some examples of these are autossh, which allows you keep a tunnel monitored and open for when you need it. Also corkscrew , which allows you to connect to remote hosts by encapsulating you traffic in https packets, thus cutting holes through proxies.

When these are combined you have the ability to connect to remote hosts, hidden behind corporate firewalls and proxies, so that you can get some work done.

Here is an example. I have a servers than can connect to all the internal network, but you can't get to this server externally, but is has access to a proxy.

Firstly you set up it's ssh config file so that it can talk through the proxy, and then you create a reverse tunnel to a machine in the cloud. You then use autossh to manage that link, so that when you need to use it , it is available.

There are so many variables in this that covering them all is not feasible, but here is an example:

First, make sure that the ssh public keys for both servers have been put into the authorized_keys file in .ssh on the other machines. Swap public keys, in other words.

Configure you .ssh/config for the user to include the following parameters
TCPKeepAlive yes
ServerAliveInterval 270
ProxyCommand /usr/bin/corkscrew 192.168.2.1 8080 %h %p

Where 192.168.2.1 is the IP of your proxy and 8080 is the port it listens on

So when we make the connection to the cloud servers, it will use this encapsulation.

Now we create the tunnel

/usr/bin/autossh -M 29001 -f -nN -R 1102:localhost:10000 billblogs@cloud01.tangerine.co.uk

Ok, lets break the command down
autossh uses -M to specify it's keep alive port, and -f to place the job in the background.

ssh uses the -nN and -R to say this is going to be a reverse tunnel, and on the cloud server it will open port 1102 and connect it to port 10000 on the internal server running the ssh daemon that we start this job from. In other words, the sshd daemon on the internal server is listening on port 10000.

So, once this is all configured, you can go to your cloud server, connect on port 1102, and it will be as if you were sat at your desk, voila

Here is the command you would use to connect:

ssh -p 1102 billbloggs@localhost

NB You need to make these connections as secure as possible, and disable root logins and only allow keys.

The sad news of the death of Christopher Hitchens

The path to Atheism for most people on the planet is complex and fraught with a great deal of inner contemplation about the big issues that face us poor scared and frightened humans. It is with the help of great writers and thinkers that this process can be crystallized into your own detailed understanding and the ability to formulate your own view of these big questions.

Now, I can't say that Christopher Hitchens was the only or indeed main influence in my change of understanding. In fact he came very late to my party of transformation , which was far more deeply influenced initially by science and specifically the evolutionary writings of people like Darwin and Dawkins.

What he did bring once I discovered his work was a writer and essayist of such great conviction and clarity of thought, that it would be worth reading his work even if you were the most bigoted religious believer. The many articles and essays that covered an enormous array of topics and subjects. The realisation that this man was not just interested in a single topic , for which he has been grotesquely pidgin holed by some corners of the media, but was in fact deeply involved in many issues , of which religiosity was just one.

One thing I'm glad to say we had in common was our love of Evelyn Waugh, and thanks to his many books and essays on George Orwell, he has encouraged me to re-engage with a writer I loved when I was much younger. His insightful coverage of the many American presidential candidates and successful incumbents was always a delight to read. His characterisations of the current crop of extreme right wing religious Republican candidates was as humorous as it was salient.

I would also encourage others to seek out his many articles and essays from magazines like The Slate, Vanity Fair, The Statesman and The Atlantic to name but a few. Each one will question your beliefs and make you think about the topic being covered.

I certainly look forward to reading more of his works on Orwell, and I would suggest dear reader that it would be profitable to yourself to do the same.

Rest in peace Mr Hitchens, you will truly be missed.

Issue with git client and Ubuntu 11.10 through corporate proxy

If like me you use git for storing code and configuration files, then it can become a problem when using it through a corporate proxy if you have the wrong versions.

I found today while trying to do a git clone the following error:

emote HEAD refers to nonexistent ref, unable to checkout

After searching around the web I came across this post.

http://goo.gl/hWhXp

This does indeed work.

Basically the problem is that the version of libcurl3 included with Ubuntu 11.10 has problem authenticating through a proxy. You either have to use 11.04 or upgrade your libcurl3 to the version in the next release of Ubuntu(precise), which fixes the issue.

I hope this saves you some effort.

This link also offers URL's to grab the latest deb packages to install, and more back ground on the issue.

http://goo.gl/XIttB

How to add another hard disk to your Xen virtual machine

If like me you have many Xen virtual machines that have been running for a while, you will likely come across the problem of having to increase the disk space, as the services or jobs the virtual machine offers over time changes.

Now you could increase the size of the existing disk, but I always feel that offers a level of risk, which with adding a new disk and using LVM or a new mount goes away.

Ok, so firstly we need to create a new virtual disk on the physical harddisk with the following command:

dd if=/dev/zero of=vm01-disk2.img bs=1G count=50

This will give us 50GB of new space. You can of course change the size to what ever you require, if your physical disk space will allow it.

Then we need to edit the Xen virtual machine configuration file and add in the new disk.

On my virtual machine host this is in /etc/xen/vm01

The line that needs to be altered is obviously the disk line, like so

disk = [ "tap:aio:/opt/xendisks/vm01.img,xvda,w",\
"tap:aio:/opt/xendisks/vm01-disk2.img,xvdb,w" ]

NB Notice the slash which allows the configuration line to expand over two lines in the configuration file.

Once you have done this, you can use the virsh and xm commands to restart the virtual machine , so that the new disk is available inside the virtual machine.

Once your VM has re-booted, you will notice another disk, which you can then either add to your LVM configuration, or just format and mount in a new directory. Here is my new mount as an example.

/dev/xvdb 50G 19G 29G 39% /opt/tomcat